Support Generic structs

[MaksimZayats]

Hi!
Is it possible to add support for generics ?

Example:

from typing import TypeVar, Generic, Any

import msgspec


T = TypeVar("T", bound=Any)


class MyStruct(msgspec.Struct, Generic[T]):
    value: T


decoder = msgspec.json.Decoder(MyStruct[int])

assert decoder.decode(b'{"value": 1}').value == 1

If I understand correctly, at the moment the only way to create a generic structure is to use type():

def GenericStruct(type_: Type) -> Type[msgspec.Struct]:
    return type(  # type: ignore
        "MyStruct",
        (msgspec.Struct,),
        {
            "__annotations__": {
                "value": type_,
            },
        }
    )


decoder = msgspec.json.Decoder(GenericStruct(int))

assert decoder.decode(b'{"value": 1}').value == 1

[jcrist]

Thanks for opening this issue. This is definitely in scope for msgspec, and is something I've been thinking about. Can you comment a bit more on your use case here?

If I understand correctly, at the moment the only way to create a generic structure is to use type():

That'd work, but we also expose msgspec.defstruct (https://jcristharif.com/msgspec/api.html#msgspec.defstruct), which should be a bit nicer to use:

import msgspec


def GenericStruct(T):
    return msgspec.defstruct("MyStruct", [("value", T)])


decoder = msgspec.json.Decoder(GenericStruct(int))

assert decoder.decode(b'{"value": 1}').value == 1

[jcrist]

Can you comment a bit more on your use case here?

@MaximZayats, can you comment more on your use case for generic structs? I'm starting to look into this, but having a motivating example would be useful.

[MaksimZayats]

@jcrist, sure.

I am currently working on fast python JSONRPC implementation with msgspec.
So I need to define base Request and Response types.
Currently I have something like this:

class Request(Struct):
    id: int | str | None = None
    method: str
    params: Raw = Raw()  # this field can be generic, but in my case `Raw` is more suitable
    jsonrpc: str = "2.0"

class SuccessfulResponse(Struct):
    id: int | str | None = None
    result: Any  # <- this field can be generic
    jsonrpc: str = "2.0"

And actually I found out, that generic structs in my case is not necessary :)
So... Maybe my examples is not really good :)

Btw, pydantic has some examples like mine:
https://pydantic-docs.helpmanual.io/usage/models/#generic-models

[tijs2]

I am looking for the same functionality because I want to inherit a base class:

It is not the nicest example but the idea is like this:

class Base(msgspec.Struct, Generic[T]):
     id: int
     value: T

class Car(Base[str]):
    extra_variable_car: int

class Bike(Base[int]):
    extra_variable_bike: int

This means I can attach new fields in the subclass and use the ones from the parent class. But also assume there is always a value field but the type could change which can be checked by Mypy if I create a certain object instance.

Is this somewhere on the roadmap to be implemented?

[jcrist]

Is this somewhere on the roadmap to be implemented?

This is definitely something that's in scope, and that I want to support. It's just not done yet. msgspec is a side hobby project for me, so features only get added when I have free time. I can't give you an estimate on when this will be resolved, but hopefully sometime in the next few months.

[Chumper]

My use case is the following.
I allow users to send messages between nodes and processes, for that I use a wrapper message to route the message to the correct process:

class BaseMessage(Struct, tag_field="msg_type", tag=str.lower):
    pass

class WrapperMessage(BaseMessage):
    to: str
    msg: T # <- this is generic

I cannot use Any, because then I cannot decode the message anymore, because the Decoder doesn't decode Any.
I could use Raw but then I need to pass the type in there as well and do the encoding/decoding myself.

I am relying heavily on type hinting to make sure only the appropriate messages are passed, so having a support for Generics out of the box would be nice.
In the end I just need to be able to pass an arbitrary message in a Struct and need to be able to decode to that in the end as well.

Last alternative for me would be to use msgspec in conjunction with cloudpickle for generic messages 🤔

[luochen1990]

Having same issue. I'm trying to define MyList like following:

class MyList (Struct, Generic[A]):
    head: A
    tail: Optional[MyList[A]]

But error occurs:

TypeError: All base classes must be subclasses of msgspec.Struct

So, what can I do to support this feature?

[gwax]

I'll give another use case where Generic structs would be very useful.

Let's say that we have some sort of tagged structures returned by an API:

class ThingOne(Struct, tag="one"):
    value: int

class ThingTwo(Struct, tag="two"):
    label: str

and the API uses a generic envelope for returning lists of tagged values such that it can be expressed as:

class ThingList(Struct, tag="list"):
    data: list[ThingOne | ThingTwo]
    total_things: int

This works fine as is but loses a lot of type information and validation power when dealing with APIs that have a known return type:

def get_thing_ones() -> ThingList:
    thing_list = msgspec.json.decode(get_data(), type=ThingList)
    return thing_list

def use_thing_ones() -> ...:
    thing_ones_list = get_thing_ones()
    thing_ones = cast(list[ThingOne], thing_ones_list.data)

With generics, especially if validated on decode, the above could be expressed as:

class ThingList(GenericStruct[T], tag="list"):
    data: list[T]
    total_things: int

def get_thing_ones() -> ThingList[ThingOne]:
    thing_list = msgspec.json.decode(get_data(), type=ThingList[ThingOne]):
    return thing_list

def use_thing_ones() -> ...:
    thing_ones_list = get_thing_ones()
    thing_ones = thing_ones_list.data

saving a lot of casting and making the return types much clearer on reading method signatures

[sanzoghenzo]

Hi, just stumbled in this exact issue...

My 2, very similar, use cases:

• need to encode some mqtt messages that share a common pattern, but a single field (let's call it "data") that holds the actual struct that varies between topics
• api responses that have pagination info and the actual data (a list of specific structs) inside a common field

Adding the support for generics would be really awesome!

[jcrist]

Support for generic structs has been merged in #386 🚀 🚀. I'm going to add generics support for dataclasses and attrs too before cutting a release (hopefully sometime in the first week of May).

If anyone is interested, I'd love it if someone could install from the main branch (https://jcristharif.com/msgspec/install.html#installing-from-github) and try things out. I believe our test suite should cover all the possible edge cases, but if there are bugs it'd be nice to catch 'em now before the release.

[gwax]

@jcrist , I can confirm that generic structs are working correctly for me in the current main branch. Thank you.