[Athen] Handwriting to OCR text

[Joshua Hori via athen-list]

And now I’ve tested with AI:

Copilot: Gives the handwritten text, nothing more.
ChatGPT: Gives the handwritten text, then explains it. The handwritten text is displayed through a code window that can be quickly copied separate from the explanation.
Claude: Gives the handwritten text but is messy and needs additional formatting. It sticks multiple bullet points into one line.
Perplexity: Gives the handwritten text and explanation but gets some of the latin font wrong.
Gemini: Gives the handwritten text and explanation, one of the better solutions.

I’ll have to test with some math equations next, then some chemistry lectures.

Oh look! An email came in and I see Aaron from Top Tech is stating the same thing. 😃

Best,

Joshua

From: athen-list <athen-list-bounces at mailman12.u.washington.edu> on behalf of Joshua Hori via athen-list <athen-list at u.washington.edu>
Date: Friday, April 11, 2025 at 1:57 PM
To: shelleyhaven techpotential.net <shelleyhaven at techpotential.net>, Karen McCall <K4mccall at outlook.com>, Access Technology Higher ATHEN <athen-list at u.washington.edu>
Subject: Re: [Athen] Handwriting to OCR text
Thanks for providing this explanation, Shelley! I appreciate the detailed comparison between handwriting OCR and dynamic handwriting recognition.

I just tested it out in OneNote, you can’t lasso handwriting in images. If an image is taken of a whiteboard with handwriting, it will not work. If you rewrite the handwritten notes, you can lasso that.

Best,
Joshua

From: shelleyhaven techpotential.net <shelleyhaven at techpotential.net>
Date: Friday, April 11, 2025 at 1:01 PM
To: Karen McCall <K4mccall at outlook.com>, Access Technology Higher ATHEN <athen-list at u.washington.edu>
Cc: Joshua Hori <jhori at ucdavis.edu>, Karlen Communications <info at karlencommunications.com>, Travis Roth <travis at travisroth.com>
Subject: Re: [Athen] Handwriting to OCR text
I've been following this handwriting OCR thread with great interest.

While not a direct answer to Lorraine's original question (converting a PDF of handwritten notes to OCR'd text), here's one other thing to consider relevant to accommodations: there's a difference between handwriting OCR and dynamic handwriting recognition.

Handwriting OCR examines the static, after-the-fact image of handwritten text or math and attempts to recognize what the user wrote by analyzing the shapes and making a "best guess" by comparing these shapes with known handwriting samples.  The process is similar to the standard OCR we know and (mostly) love.

Dynamic handwriting recognition, however, analyzes handwriting in real time (as it's being written) and as such has access to additional information with which to make decisions: the individual strokes used to create the handwriting, the sequence of the strokes, the direction of the writing utensil, the velocity of the writing utensil, pen-up and pen-down movements, and possibly the pressure of the strokes.  This produces far more accurate handwriting recognition because it's analyzing "how the image was made", not just "what the image looks like".

Dynamic HWR obviously requires a system to capture the handwriting in real time, like a digitizing tablet or touchscreen.  OneNote on a touchscreen, Livescribe smartpens, Scribble on iPad with Apple Pencil, and many other apps & devices use dynamic HWR.

So in terms of accurately recognizing and converting a student's handwriting to text or math, it is advantageous to employ an accommodation that allows the student to capture their handwriting in real time using a dynamic HWR system.

- Shelley

_____________________________
Shelley Haven  ATP, RET, BSME
Assistive Technology Consultant
www.TechPotential.net



On Apr 11, 2025, at 5:53 AM, Karen McCall via athen-list <athen-list at u.washington.edu> wrote:

Yes.

I had to charge my Surface Pen so used the touch screen to “write” a sentence. I then used the Lasso tool to select the text I’d written.

I tapped on one of the handles of the lasso “square” and got the toolbar. There is an arrow in the toolbar which I tapped. The last item in the context menu is Ink to Text.

It worked perfectly. Even with my handwriting.

Once my Surface pen is charged, I’ll try it with that.

The way the handwriting tool used to work in OneNote is based on algorithms of thousands of handwriting sample, so it is sort of like word prediction based on the database of handwriting samples. Of course, like voice recognition, it also helps you write better/more legibly as you see the common “oops, I got that wrong” results if any. So, it tends to train you instead of the other way around.

Cheers, Karen

From: athen-list <athen-list-bounces at mailman12.u.washington.edu<mailto:athen-list-bounces at mailman12.u.washington.edu>> On Behalf Of Joshua Hori via athen-list
Sent: Thursday, April 10, 2025 8:13 PM
To: Karlen Communications <info at karlencommunications.com<mailto:info at karlencommunications.com>>; 'Access Technology Higher Education Network' <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>; 'Travis Roth' <travis at travisroth.com<mailto:travis at travisroth.com>>
Subject: Re: [Athen] Handwriting to OCR text

I’ve tried it and couldn’t get it to work, so I abandoned the idea for a while. I then realized later that you need to use the lasso to convert the handwriting to text.

So…Did the lasso convert imaged handwriting to text?

Joshua


From: Karlen Communications <info at karlencommunications.com<mailto:info at karlencommunications.com>>
Date: Wednesday, April 9, 2025 at 5:40 AM
To: Joshua Hori <jhori at ucdavis.edu<mailto:jhori at ucdavis.edu>>, 'Access Technology Higher Education Network' <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>, 'Travis Roth' <travis at travisroth.com<mailto:travis at travisroth.com>>
Subject: RE: [Athen] Handwriting to OCR text
Have you checked OneNote’s ability to convert handwriting to text? Not sure it would work on a scanned graphic of both handwriting and print but might be worth a try.

I used to use it all the time for note taking. Even with my horrid handwriting, it converted accurately.

…and you’ve just given me a task to test this morning!

Cheers, Karen

From: athen-list <athen-list-bounces at mailman12.u.washington.edu<mailto:athen-list-bounces at mailman12.u.washington.edu>> On Behalf Of Joshua Hori via athen-list
Sent: Tuesday, April 8, 2025 8:59 PM
To: Travis Roth <travis at travisroth.com<mailto:travis at travisroth.com>>; Access Technology Higher Education Network <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>
Subject: Re: [Athen] Handwriting to OCR text

The camera fails and throws errors now but if you hover over it, it will still try to OCR and read it aloud. We used to use SeeingAI back in 2018 to convert handwritten notes into digital text. It worked with good accuracy back then.

This is the Image Braille GPT (free): https://chatgpt.com/g/g-fXlS5yyJA-image-braille

It’s a little challenging navigating OpenAI with a screenreader since they don’t have ARIA live tags. I find Gemini more accessible to screenreaders. NotebookLM is accessible…with exception to navigating their chat responses.

Best,

Joshua Hori
Accessible Technology Coordinator
Information Educational Technology
Academic Technology Services
50 Hutchison Dr.
Davis, CA 95616
530-752-2439
Schedule a meeting via Calendly<https://calendly.com/d/ytt-hsj-vbn>



From: athen-list <athen-list-bounces at mailman12.u.washington.edu<mailto:athen-list-bounces at mailman12.u.washington.edu>> on behalf of Travis Roth via athen-list <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>
Date: Tuesday, April 8, 2025 at 5:34 PM
To: 'Access Technology Higher Education Network' <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>
Subject: Re: [Athen] Handwriting to OCR text
Hi Joshua,
On the Read tab SeeingAI does have a take Picture button which will still show the document view. But I’ve not tried handwriting with it, so if you are saying even the take picture mode is bad at handwriting that is unfortunate.
Can you share how I may find/try the Image Braille agent?


From: athen-list <athen-list-bounces at mailman12.u.washington.edu<mailto:athen-list-bounces at mailman12.u.washington.edu>> On Behalf Of Joshua Hori via athen-list
Sent: Tuesday, April 8, 2025 7:18 PM
To: foreigntype at gmail.com<mailto:foreigntype at gmail.com>; Access Technology Higher Education Network <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>;lorraine at rcn.com<mailto:lorraine at rcn.com>
Cc: Access Technology Higher Education Network <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>
Subject: Re: [Athen] Handwriting to OCR text

SeeingAI used to convert handwriting to text, but now it just attempts to read it aloud…and it’s not as good as it used to be. That’s a shame.

You can upload an image of the handwritten text to AI and have it converted to digital text.

Gemini: Best AI for image descriptions and handwritten text.
ChatGPT: Creates great images but can go wildly inaccurate unless using the “Image Braille” GPT. That’s some high-quality system prompting powering that customized GPT.

I have video examples of using Gemini and OpenAI on my mobile device to describe lecture content in a LinkedIn article: https://www.linkedin.com/pulse/ai-musings-bridging-tactile-chemistry-lecture-audio-joshua-hori-kyglc/<https://www.linkedin.com/pulse/ai-musings-bridging-tactile-chemistry-lecture-audio-joshua-hori-kyglc/?trackingId=sKmgtHt1SPWYrhAnngJuVA%3D%3D>

Sorry, no video or audio editing tools available. I stream/record on a budget.

Best,

Joshua Hori
Accessible Technology Coordinator
Information Educational Technology
Academic Technology Services
50 Hutchison Dr.
Davis, CA 95616
530-752-2439
Schedule a meeting via Calendly<https://calendly.com/d/ytt-hsj-vbn>




From: athen-list <athen-list-bounces at mailman12.u.washington.edu<mailto:athen-list-bounces at mailman12.u.washington.edu>> on behalf offoreigntype at gmail.com<mailto:foreigntype at gmail.com> via athen-list <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>
Date: Tuesday, April 8, 2025 at 8:37 AM
To: lorraine at rcn.com<mailto:lorraine at rcn.com> <lorraine at rcn.com<mailto:lorraine at rcn.com>>
Cc: Access Technology Higher Education Network <athen-list at u.washington.edu<mailto:athen-list at u.washington.edu>>
Subject: Re: [Athen] Handwriting to OCR text

I am unaware of any OCR software that can do both handwriting and text.

Susan's advice is the best: You'll have to do this in several steps. Use a good text based OCR software (Abbyy FineReader or OmnipagePro) for the text and insert (type up the handwriting manually) separately after removing it (by cleaning up) from the original.

No easy button for this one!

Wink

On Tue, Apr 8, 2025, 7:20 AM <lorraine at rcn.com<mailto:lorraine at rcn.com>> wrote:
Wink,

It is handwriting on a PDF.  It is the other software that I am looking for recommendations on.

Thanks

Lorraine
_______________________________________________
athen-list mailing list
athen-list at mailman12.u.washington.edu<mailto:athen-list at mailman12.u.washington.edu>
http://mailman12.u.washington.edu/mailman/listinfo/athen-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman12.u.washington.edu/pipermail/athen-list/attachments/20250411/af4e26d4/attachment.html>