Method for converting non-list containers into arrays with convert

[dmckeone]

Description

This is an extension of #416 -- thanks for that. Much appreciated.

When converting from SQLAlchemy models it mostly works, except for the custom iterable type that SQLAlchemy uses, the InstrumentedList

Example:

from msgspec import Struct, convert
from sqlalchemy import select, create_engine
from sqlalchemy.orm import selectinload, Session
from sqlalchemy import ForeignKey
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
from typing import List


# SQLAlchemy 2.0 Models
class Base(DeclarativeBase):
    pass


class GrommetTable(Base):
    __tablename__ = 'grommet'

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]
    widget_id: Mapped[int] = mapped_column(ForeignKey("widget.id"))

    widget: Mapped["WidgetTable"] = relationship(back_populates="grommets")


class WidgetTable(Base):
    __tablename__ = 'widget'

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]

    grommets: Mapped[List["GrommetTable"]] = relationship(back_populates="widget")


class Grommet(Struct):
    id: int
    name: str
    widget_id: int


class Widget(Struct):
    id: int
    name: str
    grommets: List[Grommet]


engine = create_engine("sqlite+pysqlite:///:memory:", echo=False)

# Create table structure
Base.metadata.create_all(engine)

# Create demo data
g1 = GrommetTable(id=1, name="Grommet 1")
g2 = GrommetTable(id=2, name="Grommet 2")
g3 = GrommetTable(id=3, name="Grommet 3")
g4 = GrommetTable(id=4, name="Grommet 4")
g5 = GrommetTable(id=5, name="Grommet 5")
g6 = GrommetTable(id=6, name="Grommet 6")

w1 = WidgetTable(id=1, name="Widget 1", grommets=[g1, g2, g3])
w2 = WidgetTable(id=2, name="Widget 2", grommets=[g4, g5, g6])

with Session(engine) as session:
    session.add_all([w1, w2])
    session.commit()

with Session(engine) as session:
    query = select(WidgetTable).options(selectinload(WidgetTable.grommets))
    result = session.execute(query)

    for widget in result.scalars():
        widget = convert(widget, Widget, from_attributes=True)
        print(widget)

Traceback (most recent call last):
  File "/path/to/sqlalchemy-playground/parse.py", line 70, in <module>
    widget = convert(widget, Widget, from_attributes=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
msgspec.ValidationError: Expected `array`, got `InstrumentedList` - at `$.grommets`

Looks like the validation is tripping on the list-like (but not actual list) type.

If I hack in a real list in place of the Instrumented list then it works fine:

with Session(engine) as session:
    query = select(WidgetTable).options(selectinload(WidgetTable.grommets))
    result = session.execute(query)

    for widget in result.scalars():
        widget.__dict__['grommets'] = [g for g in widget.grommets]   # HACK: Convert InstrumentedList to list
        widget = convert(widget, Widget, from_attributes=True)
        print(widget)

Output:

Widget(id=1, name='Widget 1', grommets=[Grommet(id=1, name='Grommet 1', widget_id=1), Grommet(id=2, name='Grommet 2', widget_id=1), Grommet(id=3, name='Grommet 3', widget_id=1)])
Widget(id=2, name='Widget 2', grommets=[Grommet(id=4, name='Grommet 4', widget_id=2), Grommet(id=5, name='Grommet 5', widget_id=2), Grommet(id=6, name='Grommet 6', widget_id=2)])

Is there a recommended method to fix this? Is it a bug? A feature? Not sure. It does appear to be the only blocker for using convert with SQLAlchemy 2.0 model instances.

[jcrist]

Thanks for the excellent reproducible example. This has been fixed in #453. If you'd like to try the fix out before the next release, you can install the main branch from GitHub.