Add support for predicting non-binary genders

[gedankenstuecke]

Olivia pointed me to /~https://github.com/ajdavis/twitter-gender-distribution which uses the twitter bios to learn about pronoun declarations.

Pro:

• that’s the right way to do it 👍

Con:

• twitter archives don’t contain that info, would need start scraping the profiles that are mentioned in an archive. Adds runtime and dependency on twitter api.

I do want to do some research into that.

[ajdavis]

Hi! A warning that if you go down this path, the Twitter API will rate-limit you. That's why proporti.onl (which runs the twitter-gender-distribution app) asks users to log in with Twitter, just so that it can use their API budget. Even when proporti.onl does authenticated API requests on behalf of a logged-in user, there are rate limits.

If you try to regular HTML scraping instead of using the API, Twitter will detect multiple requests from the same source and rate-limit you.

So the best solution I've found is what proporti.onl does: ask the user to log in, and do authenticated API requests. For users with many followers or who follow many people, request a sample of a few hundred profiles instead of all of them.

[gedankenstuecke]

Thanks! Those are all very good points! I'm currently actually working on building an extension project in which people could log in with the Twitter API to automatically update the archives with new data going forward. That would also offer a way to implement the fetching of Twitter bios :)