Skip to content
This repository has been archived by the owner on Aug 15, 2019. It is now read-only.

Fix tf.fromPixels when using tfjs-core from node (cpu backend) #1127

Merged
merged 4 commits into from
Jun 28, 2018

Conversation

dsmilkov
Copy link
Contributor

@dsmilkov dsmilkov commented Jun 27, 2018

Fixes tensorflow/tfjs#298.

This PR along with tensorflow/tfjs-node#105 allow users to use tf.fromPixels() within node.

When in node, tf.fromPixels(pixels) now expects pixels to satisfy the HTMLCanvasElement interface, like the object returned by the canvas npm package.

BUG


This change is Reviewable

@nkreeger
Copy link
Contributor

:lgtm_strong:


Review status: :shipit: complete! 1 of 1 LGTMs obtained


Comments from Reviewable

@nsthorat
Copy link
Contributor

nsthorat commented Jun 28, 2018

FYI on Safari 2 of the unit tests are failing:

Safari 11.1.0 (Mac OS X 10.13.4) fromPixels {"BACKEND":"test-cpu"} throws when passed a primitive number FAILED
	Expected function to throw an exception with a message matching /pixels passed to tf.fromPixels\(\) must be either/, but it threw an exception with message 'pixels is of unknown type: Number'.
	<Jasmine>
	src/ops/array_ops_test.ts:1355:20 <- src/ops/array_ops_test.js:4304:36
	<Jasmine>

Safari 11.1.0 (Mac OS X 10.13.4) fromPixels {"BACKEND":"test-cpu"} throws when passed a string FAILED
	Expected function to throw an exception with a message matching /pixels passed to tf.fromPixels\(\) must be either/, but it threw an exception with message 'pixels is of unknown type: String'.
	<Jasmine>
	src/ops/array_ops_test.ts:1361:20 <- src/ops/array_ops_test.js:4314:36
	<Jasmine>

@nsthorat
Copy link
Contributor

:lgtm_strong:

LGTM once you fix the two Safari errors


Review status: :shipit: complete! 2 of 1 LGTMs obtained


Comments from Reviewable

@dsmilkov
Copy link
Contributor Author

Done. All tests pass


Review status: :shipit: complete! 2 of 1 LGTMs obtained


Comments from Reviewable

@dsmilkov dsmilkov merged commit 8e2d9c0 into master Jun 28, 2018
@dsmilkov dsmilkov deleted the frompixels branch June 28, 2018 14:02
dsmilkov added a commit to tensorflow/tfjs-node that referenced this pull request Jun 28, 2018
Implements tf.fromPixels() by allowing users to pass an object that satisfies the HTMLCanvasElement interface, as the one returned by the 'canvas' npm package. See unit test for user-code.

This PR along with tensorflow/tfjs-core#1127 fixes tensorflow/tfjs#298.
@asadovsky
Copy link

This implementation of fromPixels is still rather limiting in Node.js. For example, rather than taking on the https://www.npmjs.com/package/canvas dependency, I'd like to use https://www.npmjs.com/package/jpeg-js to decode a jpeg into a byte array plus dimensions, and pass those to fromPixels. The current code prevents me from doing that - specifically the "if (ENV.get('IS_NODE') && (pixels as any).getContext == null)" check. As a workaround, I've implemented my own version of fromPixels, but it would be nice if the official version handled such cases.

@nkreeger
Copy link
Contributor

nkreeger commented Jul 9, 2018

We are currently working on porting over the image ops from core TensorFlow to the node.js bindings. This future functionality will add more flexibility to other use cases in Node.js. Stay tuned for updates.

@land007
Copy link

land007 commented Jul 13, 2018

Thanks for this fix, I used the new version of tfjs-node@0.1.8 on the Intel(R)Xeon(R)CPU E5-2650 v4 @ 2.20GHz machine, using WebSocket to connect the node-rtsp-stream stream. Jsmpeg draws it on the node-canvas to get the 640*360 image, then uses the tensorflow model/posenet to perform the multi-person detection estimateMultiplePoses, and obtains the result of processing 1 frame in about 170-180 milliseconds. The single-node process takes up about 900% of the CPU.

@nkreeger
Copy link
Contributor

@land007 that is pretty cool - do you have any source code to share? This might be a pretty cool showcase.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants