Research Interests
Publications
Awards & Honors
Service
firstname.lastname@gmail.com
I am a research scientist at the Allen Institute for Artificial Intelligence (Ai2) and an incoming Assistant Professor at Carnegie Mellon University (CMU). I am the creator and maintainer of bitsandbytes.
I have a PhD from University of Washington advised by Luke Zettlemoyer working on efficient deep learning at the intersection between machine learning, natural language processing, and computer systems with a focus on quantization and sparsity. My main research goal is to empower everyone to make AI their own. I do this by making large models accessible through my research (QLoRA, LLM.int8(), k-bit inference scaling laws, Petals, SWARM) and by developing software that makes it easy to use my research innovations (bitsandbytes).
Previously I interned at the UCL Machine Reading Group where I was advised by Sebastian Riedel working on information retrieval and link prediction in knowledge graphs. I did my master in computer science at the University of Lugano, where I was advised by Fabio Crestani. During my master, I did two research internships at Microsoft under Chris Basoglu, where I worked speech recognition and memory-efficient deep learning algorithms. I did a bachelor in applied mathematics with The Open University while working as a software engineer in the automation industry.
In 2013, I also took part in Kaggle competitions where I peaked at world rank 63 (top 0.22%).
Feel free to contact me at firstname.lastname@gmail.com; if you have questions regarding deep learning, I prefer that you post your questions as comments on one of my blog posts (or on this page if it does not fit to any blog post); this way everyone can profit from your questions and my answers.
Research Interests
My main research thesis is that computational efficient methods will accelerate and enable progress in and understanding of deep learning. In particular, I am interested in:
- Sparse training and inference
- Mechanistic interpretability of deep learning
- Hardware optimized deep learning
- Making large models more accessible
- Low-bit quantization of large language models
Selected Publications
2023
QLoRA: Efficient Finetuning of Quantized LLMs. Tim Dettmers*, Artidoro Pagnoni*, Ari Holtzman, Luke Zettlemoyer. NeurIPS 2023 (oral). [arXiv]
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient. Max Ryabinin*, Tim Dettmers*, Michael Diskin, Alexander Borzunov. ICML 2023. [arXiv]
2022
The case for 4-bit precision: k-bit Inference Scaling Laws. Tim Dettmers, Luke Zettlemoyer. ICML 2023. [arXiv]
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer. NeurIPS 2022. [arXiv] [blog post]
Petals: Collaborative Inference and Fine-tuning of Large Models. Alexander Borzunov*, Dmitry Baranchuk*, Tim Dettmers*, Max Ryabinin*, Younes Belkada*, Artem Chumachenko, Pavel Samygin, Colin Raffel. In submission. [arXiv] [website] [chatbot]
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, Luke Zettlemoyer. In submission. [arXiv]
2021
8-bit Optimizers via Block-wise Quantization. Tim Dettmers, Mike Lewis, Sam Shleifer, Luke Zettlemoyer. ICLR 2022 (Spotlight). [arXiv] [library] [video]
BASE Layers: Simplifying Training of Large, Sparse Models. Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, Luke Zettlemoyer. ICML, 2021. [arXiv] [code]
2019
Sparse Networks from Scratch: Faster Training without Losing Performance. Tim Dettmers, Luke Zettlemoyer. [arXiv] [bib] [code] [blog post] [Q&A]
2018
Convolutional 2D Knowledge Graph Embeddings, Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel. AAAI2018. [arXiv] [bib] [code] [data] [Q&A]
2016
8-Bit Approximations for Parallelism in Deep Learning, Tim Dettmers. ICLR2016. [arXiv] [bib] [code] [data]
Awards & Honors
2023 Madrona Prize
2023 Google Open Source Award
2023 PyTorch Foundation Award
2023 Martin & Beate Block Award
2021 NeurIPS 2021 Best Reviewer Award
2018/2019 Jeff Dean – Heidi Hopper Endowed Regental Fellowship
2016/2017 Google Scholarship
Service
Reviewing:
- ICLR: 2018-2023
- NeurIPS 2019-2023
- ICML 2021-2023
- ARR 2022-2023
- JMLR 2020-2021
- IEEE Computational Intelligence Magazine (CIM) 2020
- Knowledge and Information Systems (KAIS) 2018