BEGIN:VCALENDAR
PRODID:-//AddEvent Inc//AddEvent.com v1.7//EN
VERSION:2.0
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
BEGIN:STANDARD
DTSTART:20261101T010000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=11
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
TZNAME:PST
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20260308T030000
RRULE:FREQ=YEARLY;BYDAY=2SU;BYMONTH=3
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
TZNAME:PDT
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
DESCRIPTION:Sparse attention methods promise more efficient inference and extended context capabilities for Transformer-based Large Language Models (LLMs)\, yet their practical trade-offs and scaling behaviours are not fully understood. In this talk\, I present the largest empirical evaluation of training-free sparse attention methods to date\, examining how model size\, sequence length\, and sparsity level impact their performance across diverse long-sequence tasks. I highlight key insights\, including when sparse attention is preferable over dense models\, limitations in consistently preserving performance across tasks\, and introduce novel scaling laws to predict sparse attention performance beyond tested configurations.\n\nPiotr Nawrot is a third-year PhD student at the University of Edinburgh advised by Edoardo M. Ponti and Ivan Titov. His research focuses on improving the efficiency of Transformer models. More specifically\, he is interested in teaching the models to dynamically allocate compute based on input difficulty\, which could pave the way for tokenization-free architectures and more compute-optimal models.\n\n------\n\nCreate your own Add to Calendar links with addevent.com/r/a \n
X-ALT-DESC;FMTTYPE=text/html:Sparse attention methods promise more efficient inference and extended context capabilities for Transformer-based Large Language Models (LLMs), yet their practical trade-offs and scaling behaviours are not fully understood. In this talk, I present the largest empirical evaluation of training-free sparse attention methods to date, examining how model size, sequence length, and sparsity level impact their performance across diverse long-sequence tasks. I highlight key insights, including when sparse attention is preferable over dense models, limitations in consistently preserving performance across tasks, and introduce novel scaling laws to predict sparse attention performance beyond tested configurations.<br><br>Piotr Nawrot is a third-year PhD student at the University of Edinburgh advised by Edoardo M. Ponti and Ivan Titov. His research focuses on improving the efficiency of Transformer models. More specifically, he is interested in teaching the models to dynamically allocate compute based on input difficulty, which could pave the way for tokenization-free architectures and more compute-optimal models.<br /><br />------<br /><br />Create your own Add to Calendar links with addevent.com/r/a <br>
UID:33fb5f26c5184d4daef8bf6f599c96d5addeventcom
SUMMARY:Piotr Nawrot - The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs (ML Eff)
DTSTART;TZID=America/Los_Angeles:20250528T090000
DTEND;TZID=America/Los_Angeles:20250528T100000
DTSTAMP:20261110T072355Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:0
LOCATION:https://meet.google.com/wdk-yipf-zjd?hs=122&authuser=0
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
BEGIN:VALARM
TRIGGER:-PT30M
ACTION:DISPLAY
DESCRIPTION:Reminder
END:VALARM
END:VEVENT
END:VCALENDAR