Pushed to the LLM-it
Bolus Dose EP. 05
In this episode, we discuss more of the technical aspects of LLM implementation in healthcare, including the following topics:
Embeddings
Retrieval augmented generation (RAG)
Fine tuning
Low-Rank Adaptation (LoRA)
Small models
Quantisation
Encoder and decoder models
Multimodal transformers
Some resources and papers we discuss:
Examples of RAG in healthcare: Ge J, Sun S, Owens J, Galvez V, Gologorskaya O, Lai JC, Pletcher MJ, Lai K. Development of a liver disease-specific large language model chat interface using retrieval-augmented generation. Hepatology. 2024 Mar 7.
Discussion of how concepts are changed through layers of the model in Claude: Templeton, et al., "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", Transformer Circuits Thread, 2024.
NYUtron encoder model: Jiang, L.Y., Liu, X.C., Nejatian, N.P. et al. Health system-scale language models are all-purpose prediction engines. Nature 619, 357–362 (2023).