Intent to transition from Buffer to Uint8Array

[sindresorhus]

I plan to move all my packages away from Buffer. I encourage you to do the same.

Blog post

[jimmywarting]

It will most likely eventually be supported natively in JavaScript.

How can you be so sure?

in the same repo you linked to, there is this issue:

Is encouraging binary encoding/decoding a good idea? Should it be so prominent?

I would argue that it would be best to avoid base64 and hex in the first place... stick to using arraybuffer, typedarray and blobs

[sindresorhus]

How can you be so sure?

Because the demand exists.

I would argue that it would be best to avoid base64 and hex in the first place... stick to using arraybuffer, typedarray and blobs

I don't buy that argument. Everything can be abused. Base64/Hex are useful for many things where raw bytes make no sense.

[jimmywarting]

Maybe in some places. hex and base64 could be useful, but then they should be really short.
They are no good for large data as it would be larger in size

hence why i would not "encourage" it. it could be there as an option, but i wouldn't promote it.
for instance it would be better to use URL.createObjectURL over base64 urls.
base64 are sometimes a tricky thing as they can be base64 or base64url safe which is handled very differently and might be troublesome for some... also another reason i propose to only use typed arrays.

In one of my project i replaced even a short 20byte hex (a sha1) into using uint8array all over the place in multiple packages.

imo they should only be needed if you want to present something to the user.
so visually hex is fine, but in code, well, then it should be a typed array or something.

i could say the same thing about dates. trough out everything dates should really only be timestamps, in the api, code, database and so forth, the only time it should be a string is when you need to present it to the user.
it's like the view is responsible for presenting stuff.

[sindresorhus]

Yes, I agree in principle, but reality is not as simple. For example, many APIs accept only Base64 for data, and while I do agree that's wrong, I cannot change those APIs, but I have to work with them.

[sindresorhus]

i could say the same thing about dates. trough out everything dates should really only be timestamps, in the api, code, database and so forth, the only time it should be a string is when you need to present it to the user.

That is an even bigger problem I would say. But also because it's very complex and misunderstood. At least the Temporal APIs will help guide users in a better direction.

[jimmywarting]

That is an even bigger problem I would say. But also because it's very complex and misunderstood. At least the Temporal APIs will help guide users in a better direction.

okey, maybe that was a bad example, what i wanted to get at is that "The view should be responsible for presenting the data in the preferable representation the user wants to see stuff in"
So it's better to have a date object or timestamp or even a temporal thing. and they should be handed to you and me in the same way.

A date string with year, month and date is a nightmare, you might want to view a date as "yy-mm-dd" and i might want to look at the date as "dd-mm-yy"
so it's better to always have the thing as a more low level representation that all code can understand and share in the same way across a network, code or applications. then you would use the last "date.toLocalString()" when presenting the date to user - cuz the view is suppose to use the toLocalString

So what i kind of want to get at is that base64 & hex should best be avoided unless you need to present it to the user like in a hex editor or something. the best choice would be to keep whatever data you have as a uint8array as long as it's possible. or as you say be unfortunately to work with an 3rd party api that only deals with base64

And as soon as a hex or base64 string leaves the view then it should be converted to typed arrays as quickly as possible so all other controllers, api, functions, etc works with just Uint8array.

I still have this strong View-Model-Controller (MVC) mindset back from the angular v1 time. that HTML is responsible for viewing things in the way ppl wants to see stuff. and javascript controller dose nothing in regards to sharing date strings, base64 or hex,

---

maybe this is getting a bit out of topic on the "buffer vs uint8array" thing so feel free to delete, edit, hide or whatever.

[jimmywarting]

[sindresorhus]

I have never used DataView. Do you have any example of migrating from Buffer to DataView?

[sindresorhus]

Done 👍

[PatAbah]

What did you do? Do you have an example?

[andrueandersoncs]

while Uint8Array#slice() creates an immutable copy, resulting in possible unpredictable behavior

I'm a little confused by this line, given that an immutable copy provides the opposite of "unpredictable behavior", could you expand upon what you meant here?

[sindresorhus]

If you continue reading after that sentence:

The problem is not the behavior of the Buffer#slice() method, but the fact that Buffer is a subclass of Uint8Array and completely changes the behavior of an inherited method.

The problem is not that it creates an immutable copy. The problem is that Buffer does not for the same inherited method. So if you accept Uint8Array and use slice() you could either be getting an immutable or mutable copy depending on whether the input was a Buffer or Uint8Array.

[jimmywarting]

in other word, buffer.slice() behave the same way as buffer.subarray()

buffer.slice() should never have overridden the behavior of the original .slice() method which is to create an immutable copy.
if their intention was to not copy the underlying arraybuffer, then NodeJS and everybody else should have used subarray instead in the first place and trough out all their documentation and and code examples.

b/c of this conflict buffer.slice() have now been marked as deprecated, as something bad...

[andrueandersoncs]

To be clear, my stance is that one should use Uint8Arrays rather than Buffers in all cases. My confusion with the blog post stems from the wording:

For instance, Buffer#slice() creates a mutable segment linked to the original Buffer, while Uint8Array#slice() creates an immutable copy, resulting in possible unpredictable behavior. The problem is not the behavior of the Buffer#slice() method, but the fact that Buffer is a subclass of Uint8Array and completely changes the behavior of an inherited method. Instead of Buffer#slice(), use Uint8Array#subarray() or Buffer#subarray().

The fact that "Uint8Array#slice() creates an immutable copy" directly precedes "resulting in possible unpredictable behavior" makes the sentence sound like you're saying Uint8Array#slice() causes the unpredictable behavior. I'd suggest rewording it to:

While Uint8Array#slice() creates an immutable copy, Buffer#slice() creates a mutable segment linked to the original Buffer, resulting in possible unpredictable behavior.

[jimmywarting]

even better: also mentioning that buffer.slice behaves more like buffer.subarray

While Uint8Array#slice() creates an immutable copy, Buffer#slice() creates a mutable segment linked to the original Buffer and behaving more like .subarray(), resulting in possible unpredictable behavior.

think it's also worth having

"Buffer is a subclass of Uint8Array and completely changes the behavior of inherited methods."

you may think that you are working with a TypedArray and not a node buffer if you do any instanceof uint8array or using ArrayBuffer.isView checks.

which means that using slice method is unpredictable and unsafe.