Add Proxy Support #2814

tim95030 · 2016-07-31T08:32:55Z

Description

In order to better prevent IP blocking I have implemented Proxy support for requests. It is off by default, but can be enabled by providing a proxy url such as socks5://127.0.0.1:9050. Other proxies should work as well.

Motivation and Context

Recent IP issues from Niantic

How Has This Been Tested?

The code seems to be working well at least on my Mac.

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.

Modiljani · 2016-07-31T09:13:13Z

a very reasonable offer. If you will start please inform

brandonshults · 2016-07-31T10:47:00Z

What is required beyond this pull request and the new command line flag to help with testing?

DionVermeulen · 2016-07-31T12:27:46Z

would be great!

mdsummers · 2016-07-31T14:17:25Z

pogom/pgoapi/rpc_api.py


 import protos.RpcEnum_pb2 as RpcEnum
 import protos.RpcEnvelope_pb2 as RpcEnvelope

 class RpcApi:

-    def __init__(self, auth_provider):
+    def __init__(self, auth_provider, use_tor=False):


I'm seeing an issue where use_tor is always set to false despite config['USE_TOR'] being set to true in get_args(). As a result, login never goes through the socks proxy.

chuyskywalker · 2016-07-31T14:54:44Z

Seems like this could be any kind of proxy, not specifically tor, no?

tim95030 · 2016-07-31T16:47:47Z

Yeah I suppose so. I can make it more generic. Regarding config['USE_TOR'] if you set -ut when you launch the app it definitely is working for me. So, let me know what you tried that isn't going through the proxy.

Edit: I see you are saying login isn't going through the proxy. That is possible I will look into this. I was trying to figure out how to get the settings passed through to the api and there wasn't a great way to do this. The API is created in the top of Search.py on line 35 before the args are parsed. So it is very possible some initial connections won't go through the proxy. I will see what I can do.

julien-blanchon · 2016-07-31T16:50:09Z

How to use this args ?

tim95030 · 2016-07-31T16:53:34Z

Usage:
-ut to enable the proxy
-tp to set the tor control password

You need to have tor running in the background listening on port 9050 and control port open at 9051 with a password. Here is an example torrc file that configures this:

ControlPort 9051

HashedControlPassword 16:532B9A8DEA262B0760DFDB8030CD4EACCF9A4C0D0D5C570A8954F440B5

Use tor --hash-password "password" to generate the hash. The above hash is literally "password" as the password (without quotes).

tim95030 · 2016-07-31T16:54:45Z

I am currently resetting the tor IP connection on each new search pass this is the Tor specific stuff in this pull request, but I can split that and the proxy config so any socks proxy can be used.

julien-blanchon · 2016-07-31T16:56:06Z

And for heroku ?

tim95030 · 2016-07-31T17:00:16Z

I have no idea if this will work for Heroku. I also have never used heroku. The reason I am doing this is to guard my own ip and it seems to keep my rate limit higher. I was able to run with 3 threads for the last 12 hours with no errors something I have not been able to do without this code.

tim95030 · 2016-07-31T17:02:25Z

http://stackoverflow.com/questions/10691188/use-proxy-or-tor-within-heroku-rails-app-to-hide-ip

It sounds like you would have to host tor elsewhere which would limit slow requests down in general, so I don't recommend this with Heroku.

AHAAAAAAA · 2016-07-31T17:36:37Z

I'm really conflicted about this. While this is a great solution to the recent changes, I'd like to not overwhelm the Tor network with pokemon maps without giving back. Journalists, activists, and a lot of normal people use Tor to protect themselves online in dangerous places and this would almost certainly increase latency for them for the sake of allowing us to enjoy our game more. Please consider running your own Tor relay if you do use this solution. I'd like to hear from everyone what they think about this.

mrlindstrom · 2016-07-31T17:38:43Z

I think this is a good feature to add but as you said, it would be recommended to also run your own relay.

ghost · 2016-07-31T17:59:31Z

I think need to add work with proxy, across you own server with squid.

tim95030 · 2016-07-31T18:06:56Z

@AHAAAAAAA If that is the global feeling I can remove the tor portion. I re-implemented this more as a proxy option. It works with tor, but again I can easily remove that portion.

AHAAAAAAA · 2016-07-31T18:51:12Z

@tim95030 yeah I think that would be the best way. There are other services not as resource strapped as Tor that we could use.

mcclurej06 · 2016-08-01T01:21:21Z

Would really like to see this merged. I have reason to believe my IP is being throttled. Being able to use a proxy would be great.

DionVermeulen · 2016-08-01T10:40:57Z

does it work??
i want this really bad. can i just put these files in my server and run it?
and where do i need to put my config from the server?

tim95030 · 2016-08-01T16:24:35Z

It works with socks proxies. I haven't tried it with any other proxy types however.

AHAAAAAAA · 2016-08-01T16:28:21Z

This has my +1, rebase and we can merge when another +1 comes through.

CtrlAltDefeat94 · 2016-08-01T17:36:22Z

Any documentation on how to get this to work? I'd gladly test it

tim95030 · 2016-08-01T18:36:34Z

I didn't get around to adding that. I will try to today.

Here is how you use it though:

Pre-reqs

A running socks proxy server

Config.ini Method

Add a new entry for proxy: and enter the proxy address
e.g.

proxy: socks5://127.0.0.1:9050

Command Line Method

Use either -px or --proxy with the proxy address
e.g.
$ python runserver.py --proxy 'socks5://127.0.0.1:9050'

joelcogen · 2016-08-01T19:17:58Z

Just tested on Heroku and it works. Thanks a lot @tim95030 👍

tim95030 · 2016-08-01T19:34:55Z

Added config.ini.example entry

CtrlAltDefeat94 · 2016-08-01T19:49:45Z

Not sure if I screwed something up on my side but, when I try this with Tor;
https://gist.github.com/CtrlAltDefeat94/1139d25fbc469d3aff671da73a63fc45

DomGrieco · 2016-08-01T20:58:10Z

Can this support proxies with authentication.

On Monday, 1 August 2016, CtrlAltDefeat94 notifications@github.com wrote:

Not sure if I screwed something up on my side but, when I try this with
Tor;
https://gist.github.com/CtrlAltDefeat94/1139d25fbc469d3aff671da73a63fc45

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2814 (comment),
or mute the thread
/~https://github.com/notifications/unsubscribe-auth/AGQLEkB6Vthye6okpvcAqvL2ow4pR57Xks5qbk3zgaJpZM4JY_Lg
.

Dominic Grieco
dominicgrieco.com

tim95030 · 2016-08-02T02:41:17Z

Currently this is for open proxies. I will look into authentication in a different pull request.

tim95030 · 2016-08-02T02:42:31Z

@CtrlAltDefeat94 not sure what you are doing it's saying some port is in use on the db connection. My change doesn't touch that.

CtrlAltDefeat94 · 2016-08-02T09:04:29Z

@tim95030 I start tor on a free port(tested with curl through tor that it works) and then I start the map on a free port, then I get this error :/

xzyfer · 2016-08-02T13:10:00Z

@joelcogen can you confirm how you tested this on heroku. I get exceptions with parsing the proxy flag i.e. --proxy 'socks5://some-proxy-ip:1080'

Exception in thread search_worker_0:
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
    response = self._make_rpc(endpoint, request_proto)
  File "/app/pogom/pgoapi/rpc_api.py", line 75, in _make_rpc
    r.send(prefetch=prefetch)
  File "/app/.heroku/python/lib/python2.7/site-packages/requesocks/sessions.py", line 253, in post
  File "/app/.heroku/python/lib/python2.7/site-packages/requesocks/sessions.py", line 203, in request
  File "/app/.heroku/python/lib/python2.7/site-packages/requesocks/models.py", line 481, in send
    _proxy_scheme, _proxy_host, _proxy_port = connectionpool.get_host(proxy)

joelcogen · 2016-08-02T15:20:37Z

@xzyfer tested with -px socks5://ip:port only, so short arg and no quotes

xzyfer · 2016-08-02T16:00:46Z

I tried long and short flag. Also with single, double, and no quotes. It
looks like a commit was added after your comment that may have changed
things. Can you confirm it still works with the latest version of this
branch?

On 3 Aug 2016 1:21 AM, "Joel Cogen" notifications@github.com wrote:

@xzyfer /~https://github.com/xzyfer tested with -px socks5://ip:port only,
so short arg and no quotes

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2814 (comment),
or mute the thread
/~https://github.com/notifications/unsubscribe-auth/AAjZWPJ0mLkw2Fbs9ujWuOgLYbrvxNumks5qb2BfgaJpZM4JY_Lg
.

tim95030 · 2016-08-02T16:29:13Z

Nevermind (the comment I am replacing) I had a previous instance running which is what causes the cannot bind error. As such @CtrlAltDefeat94 run ps -ef | grep python and make sure nothing is running before you try starting the server. Everybody else I am not sure what to say I have it working with a proxy both with -px and --proxy both with ' and ". I am on Mac.

tim95030 · 2016-08-02T16:33:05Z

One thing I can say is whatever proxy you are using must support https as all the calls in this api are over https

tim95030 · 2016-08-02T16:37:12Z

Here is my output:

python ./runserver.py -a ptc -u [REDACTED] -p [REDACTED] -l "[REDACTED]" -st 1 -sd 6 -k [REDACTED] --db-type mysql --db-name pokemongo --db-user pokemongo --db-pass pokemongo --db-host localhost --proxy socks5://127.0.0.1:9050 -d
2016-08-02 09:34:33,601 [      MainThread][     runserver][    INFO] Parsed location is: [REDACTED]
2016-08-02 09:34:33,603 [      MainThread][        models][    INFO] Connecting to MySQL database on localhost
2016-08-02 09:34:33,612 [      MainThread][     runserver][   DEBUG] Starting a real search thread
2016-08-02 09:34:33,613 [   search_thread][        search][    INFO] Search overseer starting
2016-08-02 09:34:33,613 [   search_thread][        search][    INFO] Starting search worker threads
2016-08-02 09:34:33,613 [   search_thread][        search][   DEBUG] Starting search worker thread 0 for user timscl
2016-08-02 09:34:33,613 [   search_thread][        search][    INFO] New location caught, moving search grid
2016-08-02 09:34:33,754 [ search_worker_0][        search][   DEBUG] Search worker thread starting
2016-08-02 09:34:33,754 [   search_thread][        search][   DEBUG] Search queue empty, restarting loop
2016-08-02 09:34:33,781 [   search_thread][        search][   DEBUG] Queueing step 1 @ 37.302033/-121.923867/0.000000
2016-08-02 09:34:33,781 [ search_worker_0][        search][    INFO] Searching step 1, remaining 0
2016-08-02 09:34:33,781 [ search_worker_0][      auth_ptc][    INFO] PTC login for: [REDACTED]
2016-08-02 09:34:34,041 [ search_worker_0][connectionpool][    INFO] Starting new HTTPS connection (1): sso.pokemon.com
2016-08-02 09:34:34,785 [   search_thread][        search][   DEBUG] Search queue empty, restarting loop
2016-08-02 09:34:34,785 [   search_thread][        search][   DEBUG] Queueing step 1 @ 37.302033/-121.923867/0.000000
2016-08-02 09:34:35,210 [ search_worker_0][connectionpool][   DEBUG] "GET /sso/login?service=https%3A%2F%2Fsso.pokemon.com%2Fsso%2Foauth2.0%2FcallbackAuthorize HTTP/1.1" 200 104
2016-08-02 09:34:35,437 [ search_worker_0][connectionpool][   DEBUG] "POST /sso/login?service=https%3A%2F%2Fsso.pokemon.com%2Fsso%2Foauth2.0%2FcallbackAuthorize HTTP/1.1" 302 0
2016-08-02 09:34:35,755 [ search_worker_0][connectionpool][   DEBUG] "GET /sso/oauth2.0/callbackAuthorize?ticket=ST-26686745-EIuEXqjtk6l1ZmUtbcZ4-sso.pokemon.com HTTP/1.1" 500 2565
2016-08-02 09:34:36,005 [ search_worker_0][connectionpool][   DEBUG] "POST /sso/oauth2.0/accessToken HTTP/1.1" 200 106
2016-08-02 09:34:36,006 [ search_worker_0][      auth_ptc][    INFO] PTC Login successful
2016-08-02 09:34:36,006 [ search_worker_0][      auth_ptc][   DEBUG] PTC Session Token: TGT-26652348-0AhLuhSERAPW
2016-08-02 09:34:36,020 [ search_worker_0][connectionpool][    INFO] Starting new HTTPS connection (1): pgorelease.nianticlabs.com
2016-08-02 09:34:38,213 [ search_worker_0][connectionpool][   DEBUG] "POST https://pgorelease.nianticlabs.com/plfe/rpc HTTP/1.1" 200 None
2016-08-02 09:34:38,215 [ search_worker_0][        search][   DEBUG] Login for account [REDACTED] successful

mcclurej06 · 2016-08-02T16:56:08Z

After some further investigation, this is unrelated to the problem I was having, but i would still like to see it implemented.

CtrlAltDefeat94 · 2016-08-02T17:16:09Z

@tim95030 after killing all python instances, the error stopped occuring(weird as none of them were using that port)
However it looks like using this PR/Tor, I am unable to use more than one instance at a time?

Is it somehow possible to still run more than one instance, as it's quite a big range that I'd like to scan

tim95030 · 2016-08-02T18:28:13Z

I am running the hex generator setting proxy: setting in the config ini and having no problem running them all concurrently. There have been some other changes around the threading as it's one account per thread now I believe. Though I haven't tried this feature. My change literally just makes all requests through the proxy it's a very simple change that has no affect on how many processes, etc. All of those things are from other features added to the project.

CtrlAltDefeat94 · 2016-08-02T19:54:56Z

Configuration error on my end, but got it working now.

+1 from me

zpv · 2016-08-03T05:34:48Z

+1 works nicely

invisiblek · 2016-08-03T10:37:19Z

This needs to be reverted. We shouldn't be touching pgoapi.

pheiberg · 2016-08-03T11:28:16Z

An idea to support socks using only environment variables in heroku: According to my limited knowledge, SOCKS5 support was added to requests in 2.10. Simply changing the requirements.txt and replacing requests==2.10.0 with requests[socks]==2.10.0 should allow setting up the HTTP_PROXY env variable to a valid SOCKS5 proxy. I successfully tried this on Heroku.

Unfortunately it fails for more than one thread.

vasilywang · 2016-08-04T17:48:36Z

@tim95030 How to contact you? I have a question about how to use your proxy code.

tim95030 · 2016-08-04T18:54:10Z

@jerry317 It was reverted. I have a pull request to the pgoapi since we no longer use it natively. Currently there is no way.

@pheiberg I in fact switched my logic in the PR for pgoapi to use request[socks] when I found out they added socks support. In theory you should be able to use what you mentioned, but you need to set two variables:
HTTP_PROXY = "socks5://host:port"
HTTPS_PROXY = "socks5://host:port"
Replacing host and port of course. We really only use https, of course until the api signing issue is fixed this is all kinda moot.

beeblebox · 2016-08-04T20:28:44Z

@tim95030 check your personal Facebook messages, it's important

tim95030 force-pushed the tor branch 2 times, most recently from c1a18b3 to bc04202 Compare July 31, 2016 09:03

mdsummers reviewed Jul 31, 2016
View reviewed changes

tim95030 force-pushed the tor branch from bc04202 to acc73f7 Compare July 31, 2016 18:05

tim95030 changed the title ~~Add Tor Support~~ Add Proxy Support Aug 1, 2016

tim95030 force-pushed the tor branch from acc73f7 to b140268 Compare August 1, 2016 00:33

tim95030 force-pushed the tor branch from b140268 to a26f776 Compare August 1, 2016 16:24

AHAAAAAAA added the has +1; needs another tester label Aug 1, 2016

tim95030 force-pushed the tor branch from a26f776 to ba0d922 Compare August 1, 2016 16:32

Allow proxy usage

cd960d5

tim95030 force-pushed the tor branch from ba0d922 to cd960d5 Compare August 1, 2016 19:34

AHAAAAAAA merged commit 8d21421 into AHAAAAAAA:develop Aug 3, 2016

invisiblek mentioned this pull request Aug 3, 2016

Revert "Add Proxy Support" #3304

Merged

tim95030 deleted the tor branch August 4, 2016 18:54

Add Proxy Support #2814

Add Proxy Support #2814

Conversation

tim95030 commented Jul 31, 2016 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Modiljani commented Jul 31, 2016

brandonshults commented Jul 31, 2016

DionVermeulen commented Jul 31, 2016

mdsummers Jul 31, 2016

Choose a reason for hiding this comment

chuyskywalker commented Jul 31, 2016

tim95030 commented Jul 31, 2016 • edited Loading

julien-blanchon commented Jul 31, 2016

tim95030 commented Jul 31, 2016

tim95030 commented Jul 31, 2016

julien-blanchon commented Jul 31, 2016

tim95030 commented Jul 31, 2016

tim95030 commented Jul 31, 2016

AHAAAAAAA commented Jul 31, 2016 • edited Loading

mrlindstrom commented Jul 31, 2016

ghost commented Jul 31, 2016

tim95030 commented Jul 31, 2016

AHAAAAAAA commented Jul 31, 2016 • edited Loading

mcclurej06 commented Aug 1, 2016

DionVermeulen commented Aug 1, 2016

tim95030 commented Aug 1, 2016

AHAAAAAAA commented Aug 1, 2016

CtrlAltDefeat94 commented Aug 1, 2016

tim95030 commented Aug 1, 2016

Pre-reqs

Config.ini Method

Command Line Method

joelcogen commented Aug 1, 2016

tim95030 commented Aug 1, 2016

CtrlAltDefeat94 commented Aug 1, 2016

DomGrieco commented Aug 1, 2016

tim95030 commented Aug 2, 2016

tim95030 commented Aug 2, 2016 • edited Loading

CtrlAltDefeat94 commented Aug 2, 2016

xzyfer commented Aug 2, 2016 • edited Loading

joelcogen commented Aug 2, 2016

xzyfer commented Aug 2, 2016

tim95030 commented Aug 2, 2016 • edited Loading

tim95030 commented Aug 2, 2016 • edited Loading

tim95030 commented Aug 2, 2016 • edited Loading

mcclurej06 commented Aug 2, 2016

CtrlAltDefeat94 commented Aug 2, 2016 • edited Loading

tim95030 commented Aug 2, 2016

CtrlAltDefeat94 commented Aug 2, 2016

zpv commented Aug 3, 2016

invisiblek commented Aug 3, 2016

pheiberg commented Aug 3, 2016 • edited Loading

vasilywang commented Aug 4, 2016

tim95030 commented Aug 4, 2016

beeblebox commented Aug 4, 2016

tim95030 commented Jul 31, 2016 •

edited

Loading

tim95030 commented Jul 31, 2016 •

edited

Loading

AHAAAAAAA commented Jul 31, 2016 •

edited

Loading

AHAAAAAAA commented Jul 31, 2016 •

edited

Loading

tim95030 commented Aug 2, 2016 •

edited

Loading

xzyfer commented Aug 2, 2016 •

edited

Loading

tim95030 commented Aug 2, 2016 •

edited

Loading

tim95030 commented Aug 2, 2016 •

edited

Loading

tim95030 commented Aug 2, 2016 •

edited

Loading

CtrlAltDefeat94 commented Aug 2, 2016 •

edited

Loading

pheiberg commented Aug 3, 2016 •

edited

Loading