Add integration with Kotlin IO multiplatform library

[sandwwraith]

This will address:
#66, #204 and #251.

kotlinx.io.core.Input/Output should replace ByteArrays and Strings in Binary and String formats, thus allowing to work with streaming API and easily convert content to either byte arrays or strings with desired charsets.

[StephenOTT]

Any timeline on this? 2019? End of 2020?

[sandwwraith]

@StephenOTT As soon as kotlinx-io is ready (currently it is under huge refactoring).

[StephenOTT]

Great. Thanks.

[DaanVandenBosch]

It seems like not much is happening on the kotlinx-io repository. Is the plan still to support the use-cases linked to by this issue (e.g. (de)serializing to/from bytes) via kotlinx-io?

[sandwwraith]

Unfortunately, kotlinx-io project is indeed postponed at the time of writing. We still have a plan to provide API to interact with Java's Input/OutputStreams in 1.2 release

[sandwwraith]

Are you sure these tickets are connected? I think Spring wouldn't depend on (Beta) kotlinx-io until it is 1.0. And Spring also can use existing Input/OutputStream integration.

[Nek-12]

Actually tried to use kotlinx-io API just now and wondering if there's any progress on this after all this time? In the meantime, do you suggest we use okio directly for now?

[sandwwraith]

There's existing integration with okio (https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-json-okio/kotlinx.serialization.json.okio/). Integration with kotlinx-io is being worked on right now and I expect it to be ready with next kotlinx.serialization release.

[Thomas-Vos]

That's amazing!

I tried out the okio integration but it is really slow. I guess that has something to do with bytes / char conversion. Will that improve with kotlinx-io?

[sandwwraith]

I tried out the okio integration but it is really slow. I guess that has something to do with bytes / char conversion.

Okio integration is usually fast when you use Source/BufferedSource directly as e.g. network source of bytes. Do you have an example where it is slow?

[xiaozhikang0916]

Noticed that the related PR is just adding integration with kotlinx.io for JSON format only. Will you continue adding integration for the core lib so that every format can get benefits from streaming en/decoding?

[sandwwraith]

If there is a significant demand for Protobuf or Cbor, we can add kotlinx-io to them as well. It is also easy to add corresponding signatures to StringFormat/BinaryFormat; yet, the only reasonable default implementation for, e.g., BinaryFormat will be return Buffer().write(encodeToByteArray(..)), which involves additional bytes copying and may be not so fast. So ultimately, to efficiently use kotlinx-io, the format itself should use it.

[pdvrieze]

@sandwwraith Given that the default implementation would be clearly making things worse I would think that it is better to have sub-interfaces that formats that actually properly support it can implement. At least that means that the it is clear when it is or is not supported. Especially as the whole point is to support more efficient reading/writing.

[xiaozhikang0916]

Giving such default implementation does not help. The key classes in Json for streaming e.g. InternalJsonReader/Writer and related apis decodeByReader/encodeByWriter are just defined in Json format. It would be some boilerplate for other formats to support streaming. Actually we can see that ByteArrayInput/Output in Cbor have some similar codes with those in ProtoBuf, which we can make some abstraction to. For example, a common ByteArrayInput for all BinaryFormats to read from in member function decodeFromByteArray. By doing so, it is quite easy to add streaming support by wrapping this common ByteArrayInput with kotlinx.io, just like what json-kxio did.

[sandwwraith]

ByteArrayInput/Output are highly specific classes. E.g. protobuf.ByteArrayOutput has writeLong function but cbor.ByteArrayOutput has not. Making them public would require designing them for needs of every binary format out there, and essentially trying to invent a new IO primitive (which we already did with kotlinx-io). So the only option I see here is to eventually make kotlinx-io Source/Sink the default choice, and to make byte array adapters that would simply consume/write to it fully. It is the same non-ideal situation as before, but I think for real-world applications (writing to file, sending to network), IO-first implementation is preferable.

[sandwwraith]

Or alternatively, provide an implementation of Sink/Source that writes directly to ByteArray instead of a buffer.

[pdvrieze]

Btw. one missing bit is a common way to deal with encoding (think InputStreamReader). This functionality is needed for most textual formats (esp when they need to support non-utf encodings).

[sandwwraith]

@pdvrieze kotlinx-io may eventually support different charsets, although this task is currently not in priority

[sandwwraith]

Fixed by #2707