-
Notifications
You must be signed in to change notification settings - Fork 950
Fix tf.fromPixels when using tfjs-core from node (cpu backend) #1127
Conversation
Review status: complete! 1 of 1 LGTMs obtained Comments from Reviewable |
FYI on Safari 2 of the unit tests are failing:
|
LGTM once you fix the two Safari errors Review status: complete! 2 of 1 LGTMs obtained Comments from Reviewable |
Done. All tests pass Review status: complete! 2 of 1 LGTMs obtained Comments from Reviewable |
Implements tf.fromPixels() by allowing users to pass an object that satisfies the HTMLCanvasElement interface, as the one returned by the 'canvas' npm package. See unit test for user-code. This PR along with tensorflow/tfjs-core#1127 fixes tensorflow/tfjs#298.
This implementation of fromPixels is still rather limiting in Node.js. For example, rather than taking on the https://www.npmjs.com/package/canvas dependency, I'd like to use https://www.npmjs.com/package/jpeg-js to decode a jpeg into a byte array plus dimensions, and pass those to fromPixels. The current code prevents me from doing that - specifically the "if (ENV.get('IS_NODE') && (pixels as any).getContext == null)" check. As a workaround, I've implemented my own version of fromPixels, but it would be nice if the official version handled such cases. |
We are currently working on porting over the image ops from core TensorFlow to the node.js bindings. This future functionality will add more flexibility to other use cases in Node.js. Stay tuned for updates. |
Thanks for this fix, I used the new version of tfjs-node@0.1.8 on the Intel(R)Xeon(R)CPU E5-2650 v4 @ 2.20GHz machine, using WebSocket to connect the node-rtsp-stream stream. Jsmpeg draws it on the node-canvas to get the 640*360 image, then uses the tensorflow model/posenet to perform the multi-person detection estimateMultiplePoses, and obtains the result of processing 1 frame in about 170-180 milliseconds. The single-node process takes up about 900% of the CPU. |
@land007 that is pretty cool - do you have any source code to share? This might be a pretty cool showcase. |
Fixes tensorflow/tfjs#298.
This PR along with tensorflow/tfjs-node#105 allow users to use
tf.fromPixels()
within node.When in node,
tf.fromPixels(pixels)
now expectspixels
to satisfy theHTMLCanvasElement
interface, like the object returned by thecanvas
npm package.BUG
This change is