Search with R18.dev doesn't work anymore

[darksider59]

Hello,

Sine yesterday, the search with R18.dev doesn't work anymore. I tried identifying multiple movies, but the result is always empty.

[pplusp]

Can confirm, this is also happening to me. I'll get results from AV-Wiki, 141Jav, and OneJav most of the time, but the connection to R18 seems to be completely broken.

[darksider59]

As I'm not a dev, I got one working on the agent to fix the errors. In the end, I'll get my own agent.

[alxpnt2]

I'll take a look this weekend and see what's going on with R18

[misterhon]

It looks like the SEARCH_URL needs to be updated - r18.dev no longer uses "dvd_id" as the search parameter but just "id".

[alxpnt2]

This is the javascript code for R18.dev's search bar. It uses "dvd_id" to find the content id, then plugs that into the "id" parameter in a second request, which is how the plex agent does it as well. The issue is that all requests from plex to R18.dev are returning a 403 error, which indicates that they're being blocked. I've tried messing around with the request headers, but I haven't had any luck yet bypassing the block. If anyone has any ideas, I'm all ears.

[misterhon]

Does API_URL still work?

[alxpnt2]

It does not, every url on the jav18.dev domain returns a 403.

[ldjb]

Hello from R18.dev. One of your users pointed me to this issue.

First of all, I should stress that using R18.dev's private APIs is not supported. This is not a telling off, but rather a heads-up that you use it at your own risk. The APIs are subject to change or be withdrawn at any time, without warning. The only supported means to programmatically access R18.dev data is via the database dumps. You are free to use this data however you like, including creating your own APIs, if you so wish.

That aside, it looks like PlexJav18.bundle is being blocked by Cloudflare's built-in Bot Management feature. I notice it reports as being Google Chrome via the User-Agent string defined at:
/~https://github.com/alxpnt2/PlexJav18.bundle/blob/master/Jav18.bundle/Contents/Code/site.py#L9

Although I do not recommend using R18.dev's private APIs, if you absolutely must do so, please consider using a more descriptive user-agent string, e.g. PlexJav18.bundle, at least when making requests to R18.dev. If that in itself doesn't work, I can try unblocking it, but I really need you to use a more descriptive user-agent string to do so.

For the past several months, R18.dev has been inundated with requests that has resulted in a lot of downtime. To keep the service online, it has been necessary to implement a rate limit. Looking at the requests coming in with that particular user-agent string, I'm seeing certain clients making hundreds of requests per minute, which I would deem to be excessive.

Now, I don't know whether these are coming from PlexJav18.bundle or some other software that uses the same user-agent string, but I just wanted to make you aware that if you make too many requests in a short space of time, you may face a temporary block.

It doesn't do anyone any good if R18.dev goes down, whether that's people browsing the R18.dev website or people using software such as this. Therefore, the priority will need to be maintaining the availability and performance of R18.dev, and I reserve the right to implement whatever measures are necessary to do so.

[alxpnt2]

Thank you so much for the insight ldjb, I appreciate you reaching out. I certainly wouldn't want this plugin to have an adverse affect on R18.dev itself; it's one of the only good sources for this kind of data, and I don't want to cause you guys any trouble. I pushed an update changing the user-agent string when connecting to R18.dev (now "PlexJav18") and reduced the maximum potential number of calls to your site per title. I also disabled the plug-in connecting to your site by default, for users that download the update at least.

I would like to reduce the dependency on R18.dev's private API, but ultimately this plug-in was designed to scrape R18 when that was still around, and other sites were only added to fill holes in R18's coverage (and later cover everything, after R18's demise). As far as I know, there is no public API for this data, and creating one is a time/money expense I'm not interested in right now. I'm looking into potentially making some software that could run locally, downloading the R18.dev database dump and providing data to the plug-in, but I'm not sure when I'll have that ready.

In the meantime, ldjb, thanks for your efforts at R18.dev, and I'm sorry for the trouble this plug-in caused. If the traffic must be blocked for the health of the site, then there's no getting around that. I'll try to find another solution.

[JustynLim]

Does this mean r18.dev is still blocked even with the latest implementation? I've tried using my own user agent but I'm still getting no results from the site.

[ldjb]

I saw that PlexJav18 was still being blocked as Cloudflare was flagging it as an AI bot.
I've put in an exception for PlexJav18, so it should be working now. I'll keep an eye on it.

[JustynLim]

Could you check whether cloudflare is still blocking requests even after adding the exception?

[ldjb]

Looks like my earlier configuration didn't work, so I made another change and the requests are no longer being blocked.

For reference, turns out to bypass Cloudflare's "Manage AI bots" rule (which is enabled through their Block AI Bots feature), you need to put a skip on "All Super Bot Fight Mode Rules", not "All managed rules", despite the fact that the block appears in the logs as coming from the "Managed rules" service, and despite "Block AI Bots" being a separate option from "Bot Fight Mode" (which itself is a bit different from "Super Bot Fight Mode"). 🙄

Anyway, it should all be good now. Let me know if there are still issues.

[JustynLim]

So far there seems to be no issues fetching metadata. I've edited the user agent string to make it unique 'Justyn-PlexJav18-agent'. I've only made 2 queries from plex. You should be able to track if its making excessive calls through the API.

Thanks for helping the community out.

[pplusp]

So far there seems to be no issues fetching metadata. I've edited the user agent string to make it unique 'Justyn-PlexJav18-agent'. I've only made 2 queries from plex. You should be able to track if its making excessive calls through the API.

Thanks for helping the community out.

Where would you edit that string? Is that in the Info.plist?

[JustynLim]

So far there seems to be no issues fetching metadata. I've edited the user agent string to make it unique 'Justyn-PlexJav18-agent'. I've only made 2 queries from plex. You should be able to track if its making excessive calls through the API.

Thanks for helping the community out.

Where would you edit that string? Is that in the Info.plist?

No, it's in site_r18dev.py under the variable: R18_DEV_HDR

[SlippedItUp]

I've downloaded the new version and put it on both of my servers. Also edited the agent string to a unique one for me. It's failing to correctly add metadata... but after I disabled everything but R18.dev as options and then checked the logs, they're giving me back 200 responses, not 403s. So I have to assume the issue is somewhere on my end. I'll keep looking into it. Everything now looks fine from the R18.dev end though, so thanks to everyone for working through that.

[JustynLim]

I've downloaded the new version and put it on both of my servers. Also edited the agent string to a unique one for me. It's failing to correctly add metadata... but after I disabled everything but R18.dev as options and then checked the logs, they're giving me back 200 responses, not 403s. So I have to assume the issue is somewhere on my end. I'll keep looking into it. Everything now looks fine from the R18.dev end though, so thanks to everyone for working through that.

Maybe I could help? Could you provide me with the movie code?

[SlippedItUp]

I've downloaded the new version and put it on both of my servers. Also edited the agent string to a unique one for me. It's failing to correctly add metadata... but after I disabled everything but R18.dev as options and then checked the logs, they're giving me back 200 responses, not 403s. So I have to assume the issue is somewhere on my end. I'll keep looking into it. Everything now looks fine from the R18.dev end though, so thanks to everyone for working through that.

Maybe I could help? Could you provide me with the movie code?

SONE-413 and EUUD-59. I actually have my media split between two different servers, one on a Windows box and one on TrueNAS Scale (entirely because the latter is running low on space so I can't move everything from the former over yet), and these two are split between them. But both movies are viewable if you just go to r18.dev in a browser and manually look, so I figured they'd be a good test to see if the updated plugin works on both servers. Neither return any matches in Plex, even with only Enable/Search R18.dev enabled as options on both sides and all the other stuff, i.e. 141Jav OneJav etc, turned off.

But when I check the plugin logs of the server on Windows, it is correctly doing a GET on https://r18.dev/videos/vod/movies/detail/-/dvd_id=sone413/json, so the plugin is absolutely seeing the data at the site, I'm just not sure why it's not then matching based on it.

[SlippedItUp]

Update: I shut the servers down, deleted the agent and its cache on both sides, restarted the servers, emptied trash/cleaned bundles/optimized db, shut the servers down again, put a fresh copy of the agent on both servers, and restarted them. Things are sort of working now, with caveats. I tested this time with a newer video, SONE-548, which is definitely at r18.dev. It matches up on the AV-Wiki entry, but you can see in these logs that though it correctly figures out https://r18.dev/videos/vod/movies/detail/-/dvd_id=sone548/json as an address to pull metadata from, it then claims it can't find that page. I left the user agent as default initially, and then changed it to "SlippedItUp-PlexJav18" to differentiate things, but same result both times.

com.plexapp.agents.jav18.log

[ldjb]

Hi @SlippedItUp. It looks like Cloudflare's Bot Fight Mode service was issuing you with managed challenges for some reason. I've made a couple of changes so hopefully your requests will succeed now.

[SlippedItUp]

Hi @SlippedItUp. It looks like Cloudflare's Bot Fight Mode service was issuing you with managed challenges for some reason. I've made a couple of changes so hopefully your requests will succeed now.

It's working like a charm now, thank you so much!

That's not something you're gonna have to do individually for people though, is it? Don't want to put more load on you or the site than is necessary, for sure.

[ldjb]

I'm trying to not do things on an individual basis but have a general configuration that is effective at keeping demand manageable. The rate limit helps a lot, for instance. I think I've got it in a good place now, so hopefully won't need to make further config changes, but I'll keep an eye on performance and adjust if necessary.

[Javsearcher]

Yesterday it worked great after a few weeks not working, but today r18.dev disappeared again.

which means no picture of the actors
SSNI228 wont change to SSNI-228
And no series are filled in.

[JustynLim]

Yesterday it worked great after a few weeks not working, but today r18.dev disappeared again.

which means no picture of the actors SSNI228 wont change to SSNI-228 And no series are filled in.

I experienced the same issue when I tried matching the same code you've mentioned and saw that the agent wasn't doing its work automatically. However, it was able to find the movie without any issues if I perform a manual search on it.

The strangest thing is that once I've done so, the agent started working as expected (showing results when you click on the 'Match' option). I have no idea why older versions of Plex are still experiencing issues with automatic matching. I'm using PheonixAdult's plugin for my non-JAV content which made me aware of this issue in the first place, forcing me to manually lookup each scene

[Javsearcher]

That's what i did yesterday every movie i had to scan twice
First time it finds the file but not as i wanted it to be and the second time i scaned the agent uses the correct one (i can live with that :-) (even though a burden for the receiver to get requests twice)

but not today

even manual it doesnt find r18.dev (i tried other movies old an new ones same result

DRPT-081 new movie
jul-003 old movie

rebooten plex didnt help
Plex version 1.41.3.9314 (synology (no docker)

[JustynLim]

Is there anyway you could verify that you're actually using the latest version? It sounds like your requests are getting blocked possible due to too many requests made (if its still using the old method of getting info).

On a side note, could you provide the 2 codes you've shown in the picture (both from the bottom left).

[ldjb]

Unfortunately, R18.dev has been experiencing a DDoS attack from 10,000s of IP addresses today, so it has been necessary to increase security measures for the time being.

[Javsearcher]

Is there anyway you could verify that you're actually using the latest version? It sounds like your requests are getting blocked possible due to too many requests made (if its still using the old method of getting info).

On a side note, could you provide the 2 codes you've shown in the picture (both from the bottom left).