(I) Objective: The Medical Agent assists doctors by collecting structured patient information before the consultation. The Medical Agent asks questions about the patient’s recent symptoms, medical history, lifestyle habits, medication usage, and family history, then summarizes and structures the information to help doctors shorten interview time.

(II) Data Collection: 

• • Collect internal hospital documents, including:

    • Standard medical interview templates.

    • Symptom descriptions and red-flag cases.

    • Lifestyle and chronic-disease inquiry guidelines.

    • Family medical history question flows.

  • These documents are used to refine a medical assistant conversational agent capable of handling diverse patient conversation scenarios.

(III) Medical Agent pipeline: 

• • Patient Intake & Identity Confirmation:

    • Ask for patient identity, age, known diagnoses, ongoing medications.

    • Confirm reason for visit.

  • Dynamic Question Flow:

    • Agent asks questions in a structured and adaptive sequence: recent medical condition, current symptoms, onset time, severity, triggers.

    • Lifestyle Habits: smoking, drinking, diet, sleep, exercise.

    • Past Medical History: chronic diseases, surgeries, allergies.

    • Family Medical History

  • Speech Question Generation (TTS):

    • Convert questions into natural, clear speech.

    • Adjust tone: slow mode / elder mode / child-friendly mode.

  • Speech Check:

    • Use STT to transcribe the TTS audio

    • Compare text vs. expected question

    • If mismatch → regenerate TTS

    • Difficult word pronunciation handling:

      • Replace difficult word with same-pinyin easy word.

      • Regenerate TTS until pronunciation is acceptable.

  • Patient Answer Collection:

    • Patient gives spoken replies

    • STT transcribes full sentences

    • Use Pinyin correction + LLM to handle: unclear articulation

    • reference : https://sites.google.com/view/chihyu/main/sensevoice.

  • Structured Data Outputs: 

    • LLM converts raw conversation into structured medical notes and provides JSON output for EMR (Electronic Medical Record) integration.

  • Summary to Doctor:

    • LLM generates a concise summary

(IV)  Results: The Medical Assistant has been successfully deployed and tested in the Veterans General Hospital.