[Solved] JulesJordan.com Un-feedables

[leroyeur]

I have copied every single link in the "Scenes" section on JulesJordan.com
There are 2487 scenes in that section
When adding them in JD2, I would only end up with 2113 videos/links.
374 links are un-addable for some reason.

**External links are only visible to Support Staff**Here are 218 un-addable links

**External links are only visible to Support Staff**Here are all the 2487 links

**External links are only visible to Support Staff**Here are the 2113 that have been parsed/added successfully

I realize that there's a discrepancy between the 218 I've sent and the 374 I said weren't added, but I couldn't figure out how to get the 156 links left. I used both WinMerge and BeyondCompare to compare the entire website's links and the links that were parsed, and for some reason they only show me 218

I also realize that the log of me adding all the links would be too large, so here's a log of me only adding the 218 links

Spoiler:

09.02.22 15.20.02 <--> 09.02.22 15.21.06 jdlog://3305925302851/

I have to also add that this isn't the first time something like this happens from Jules Jordan. I have downloaded back in December, adding about 800 links, and there are many that weren't added, but I didn't keep track of them unfortunately. Same in January. But this time I have this at least.

I must also mention that sometimes adding working links doesn't work either. So I'd paste them in JD2, but nothing happens. Sometimes I'd try multiple times consecutively, and it eventually adds something, but the link is either corrupt or doesn't download, or something gets in the way.

[leroyeur]

Another problem I am currently facing is that downloads are not starting.
I have successfully downloaded entire channels from Xvideos and other websites, and they were being downloaded at a good unstoppable rate.
However, these past few days, the time between a download finishing and the next download in queue starting (from the same website) is taking too long.

And for some reason, Jules Jordan stops for long periods as well

Here is a report with 3 downloads showing errors from Jules Jordan

Spoiler:

10.02.22 12.58.56 <--> 10.02.22 13.02.02 jdlog://4805925302851/

[leroyeur]

One more update. This is the same thing that happened in December and January. If I deleted a link and re-added it in the linkgrabber, it would download. Otherwise, I'm getting an error in all added downloads when they are started.

Here's a log where I got an error for every link I tried to download (that already exists in the download queue), and then deleting a link, and re-adding it in the linkgrabber, then starting the download, and it works.

Spoiler:

10.02.22 17.41.10 <--> 10.02.22 17.48.29 jdlog://1015925302851/

[pspzockerscene]

Hi again,

1. Un-addable links:
Websites may change at any point of time, rendering our crawlers or parts of them useless.
This is most likely what has happened here.
I will need your julesjordan.com login credentials to work on this.
Please send them to us via: support@jdownloader.org.

2. Downloading issues:
Please wait for the release of our next CORE-updates and try again.
Please collect all URLs of failed downloads and provide them separately.

3. Downloads which fail and only work after re-adding them:
Please wait for the updates mentioned in your other threads here:
https://board.jdownloader.org/showthread.php?t=89540
and here:
https://board.jdownloader.org/showthread.php?p=496496

Please do not open new threads until we've released these updates.
Sorry for the delay.

-psp-

[leroyeur]

Hello, I sent you the credentials.

You mentioned that I shouldn't make more threads, but I wanted to share some more un-addable links from other websites from your plugins, if you want them.

And I'm sorry, but I don't have another copy of the document where I found the un-addable links, I only pasted them in the website I attached to the post. If I could take another look at it, I would send it again separately.

Lastly, I've been facing problems with download rate consistency and continuance. I have set JD2 to download simultaneously from multiple hosts (1 download per host), and set the max sim downloads to 20. I only have 3 hosts, and they all keep... "freezing".

I downloaded over 20,000 downloads from Xvideos from multiple user pages, but with the Xvideo downloads I have queued now, they only download very few downloads per day (it used to download consecutively, without stopping, download after download, whereas now it takes over 10 minutes to start a download after finishing the previous download from Xvideos alone).

It's not only Xvideos either. Jules Jordan would download 10s of downloads after each other, then I'd find it frozen or stuck at "starting download", or I'd see that it hasn't started the next download even though there's no error or anything.

Same with AdultTime, where I've successfully downloaded thousands of downloads before. But now for some reason it would only download a few downloads before stopping. I have to come to my computer every few hours to right click > resume download (even though the downloads aren't paused or skipped or anything, they're neutral)

So, where should I post this? Or will this be covered in the next core update? Because it wasn't an issue before, it's relatively new.

[pspzockerscene]

Originally Posted by leroyeur

Hello, I sent you the credentials.

Got them and will look into it once I find the time.

Originally Posted by leroyeur

You mentioned that I shouldn't make more threads, but I wanted to share some more un-addable links from other websites from your plugins, if you want them.

What I ment was more like "not more threads about same/similar issues".
Sure you can create more threads regarding other plugins/issues but the last one you've created was more a duplicate though we'll only really know that once you can provide further feedback after the release of our next set of updates.

Originally Posted by leroyeur

Lastly, I've been facing problems with download rate consistency and continuance. I have set JD2 to download simultaneously from multiple hosts (1 download per host), and set the max sim downloads to 20. I only have 3 hosts, and they all keep... "freezing".

Please post a log of that and/or explain how to reproduce it.
How many links are in your downloadlist in total?

Originally Posted by leroyeur

I downloaded over 20,000 downloads from Xvideos from multiple user pages, but with the Xvideo downloads I have queued now, they only download very few downloads per day (it used to download consecutively, without stopping, download after download, whereas now it takes over 10 minutes to start a download after finishing the previous download from Xvideos alone).

This could have multiple causes one of then being a kind of "shadow ban" by xvideos - I'm pretty sure that downloading so many items will trigger some kind of anti bot protection on their side.
Either way it's another topic and you're free to open a separate thread regarding this xvideos.com issue.
...though as explained it might not even be a JD bug but more rate-limiting by that host or some kind of other issue.
You might as well just post everything in this thread for now as long as it's somehow related and doesn't cause too much confusion

Originally Posted by leroyeur

It's not only Xvideos either. Jules Jordan would download 10s of downloads after each other, then I'd find it frozen or stuck at "starting download", or I'd see that it hasn't started the next download even though there's no error or anything.

Again:
With everything related to julesjordan.com and other websites for which I changed plugins (e.g. adulttime.com) after your reports:
Please wait for our next CORE-update before issuing further reports.

Originally Posted by leroyeur

So, where should I post this? Or will this be covered in the next core update? Because it wasn't an issue before, it's relatively new.

See the other parts of this reply.

You can keep reporting stuff but maybe always include a log so I can look into it.
Keep in mind that for all websites which require you to be logged in, I will need your credentials in most of all times to work on them.

Also, best would be instructions on how to reproduce bug XY consistently as this way we can find/fix it easier.

-psp-

[leroyeur]

I'll report back to you once I know how to recreate the problem. I can't send you a log because it would be too large, as I usually find the problem during my regular sessions (and my regular sessions are very long).
So I'll get back to you once I find a pattern of when this problem happens (which I've been trying to figure out for a while).

Right now, I'll disable Xvideos and keep it for later, because I want to focus on downloads from subscriptions I'm paying for, so I'd finish them before I have to pay for them again.

However, I wouldn't mind exploring the problem further now if anyone else in the community is facing it, I'd be happy to help the community.

Thanks for the response.

[pspzockerscene]

Hi again,

julesjordan.com:
I've found the reason why some links fail:
I haven't tested a lot of links but I immediately noticed that when accessing any of your "218 un-addable links" via browser, I get an extra security prompt:

New Device or Location Detected!
We haven't seen you accessing the site with this device or from your current location before.
To protect your account, we've sent you an email with an activation code to the email address you used when joining the site.
Please check your email, then enter the code you received in the form below and press "Continue" to access your videos.

Their system is kinda strange because they already let me login with my IP but this keepds happening for some content only(?)
Maybe it's even a bug on their side.
Anyways:
I've partially added support for this though I was unable to test it as I do not have access to these E-Mails.
If you want me to be able to fully test this, please add an auto e-mail forward of these mails to: psp@jdownloader.org.

Also you will probably get A LOT of these "Password" dialogs after the next update -> Once I'm able to fully test this, I will try to optimize it to e.g. only ask once per session.

I've also tested the auto-refresh of expired direct-URLs and it appeared to be working fine here.

xvideos.com & xvideos.red:
This might be unrelated to your report but I'll add this information anyways:
The following changes will be released with our next set of CORE-updates:
- fixed crawling of xvideos.red profiles
- fixed channel crawler
- removed unnecessary sleep statement before download starft: downloads will now start quicker

For all code changes mentioned in this post, our information abotu updates applies:

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[leroyeur]

Forwarding added.
Do you get the same security prompt with other addresses? I've gotten that prompt before, but it's never stopped me (or JD) from downloading, so I'm not sure how they work exactly.

I must add; sometimes I'd find downloads stopping in JD. So I'd sign in using my browser. And when I sign in, the downloads continue. I don't know what's up with that, but there is a direct tie... in some occasions.

Great work on Xvideos, thank you.
I honestly, absolutely, cannot wait for the next updates.

Thank you

[pspzockerscene]

Originally Posted by leroyeur

Forwarding added.

Confirmed working.

Originally Posted by leroyeur

Do you get the same security prompt with other addresses?

It's hard to say.
I checked 1-2 of your "un-addable" URLs and that's when I got the prompt.
I confirmed it once in my browser and then it was gone.
--> This means that they do this by IP and not by cookies which simplifies things a bit.
I've now implemented it properly in JD so if it ever happens (after the next CORE-update), you will get a prompt to enter it.

Originally Posted by leroyeur

So I'd sign in using my browser. And when I sign in, the downloads continue.

Maybe a new sign in also counts as a security confirmation, maybe you had a different issue than me - I can't really say at this moment.

Originally Posted by leroyeur

I honestly, absolutely, cannot wait for the next updates.

Same for me

So now I've made two more tests with julesjoran:
1. Added all of your "un-addable" URLs with zero failures.
(Needed security confirmation for the first one.)

2. Added your package of "2487 links" and stopped after some time (it was still crawling):
Result (with the first "218 un-addable links"):
2485 packages
18135 links

I haven't tested downloading as I simply do not have the time for this.
I recommend testing with max. 1 chunk and 2-4 simultaneous downloads and check if you still get the "freezes" then.

-psp-

[pspzockerscene]

CORE-Updates have been released!
Please update your JDownloader and report any issues you find asap.

CCORE-Updates wurden released!
Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden.

-psp-

[leroyeur]

Wonderful wonderful job
I just updated JD too. You guys... what a job. What an excellent job

[pspzockerscene]

Thanks for your feedback
I've marked this thread as [Solved] for now.
I'll leave it open so you can post further issues here if you find more.

-psp-

[leroyeur]

Guys, in my experience, when the website (JulesJordan.com) asks for the activation code sent via email, it only does it once per session. I could enter hundreds or thousands of pages before being prompted again.

Whereas with JD's new update, it is asking me for the code every time I paste (specific) Jules Jordan links. I am not even getting as many emails as it is asking me. And it tells me "Wrong Password!" when I enter the previous code, or click continue/cancel.

[Jiaz]

@leroyeur: nothing has changed with account login in plugin since month but JDownloader does not yet support *activation code sent via email* and that's most likely the cause of it.
We need username/password to check/add support for it. Can you contact us via support@jdownloader.org

[leroyeur]

Credentials sent

**External links are only visible to Support Staff**Here is a list of all the links I have. Trying to add them would prompt a code for most of the videos. I already have you guys added to email forwarding for any code.

You don't have to add all the links, you should notice very briefly after adding the first batch of links

[Jiaz]

@leroyeur: email forward did before or new? I've seen a forward request by google mail which failed to finished

[pspzockerscene]

@leroyeur
Please re-send your julesjordan.com login credentials to us via E-Mail or re-link your ticket-ID as I'm unable to find it at this moment.
It looks like the forwarding of activation codes to my email does still work - I got some mails from your download attempts some hours ago.

-psp-

[pspzockerscene]

Originally Posted by pspzockerscene

@leroyeur
Please re-send your julesjordan.com login credentials to us via E-Mail or re-link your ticket-ID as I'm unable to find it at this moment.

Found it.

I was unable to reproduce your issue thoiugh I've only tested with some thousands of links.

Please keep in mind that most of all websites will have some kind of limits for heavy users.

Here is what you might be able to do:
1. Add less links (add time-gaps between adding links).
and/or:
2. Create an EventScripter script that automatically scrapes- and inputs the "Activation code" you get via E-Mail.
This will probably be some work but this way you could let it run 100% automatically and it seems like it could be worth it for you as you do want to download a lot of content from that website.

-psp-

[Jiaz]

@leroyeur: are you using vpn/proxy in browser or JDownloader?

[leroyeur]

@pspzockerscene I haven't downloaded a single video in over a month, so I don't think it's related to any kind of restriction or special behavior towards heavy users.

I have downloaded 2133+ videos which is about 85% of the videos. They are the only ones that were able to be parsed, until you guys fixed it, thank you, so I'm only going to download the 320+ videos left, and I am only adding all the links to download the difference, but I still haven't downloaded a single video since over a month ago. I'm not sure where the requests are coming from

@Jiaz I am using a VPN for my entire Windows system (same IP for all apps). Is that an issue? Or does that change things?

[Jiaz]

@leroyeur: What VPN are you using? Have you verified that every traffic is correct tunneled? Maybe there is some misconfiguration and browser and JDownloader use different IP and thus different locations. Maybe we can do a teamviewer session together to check it out together/what is going on? contact us via support@jdownloader.org

[pspzockerscene]

Also you could simply turn off that VPN and test again to see if these issues still occur.
A lot of websites will try to block VPNs and/or you will run into additional security checks because typically for paid VPNs, a lot of different users will be running around with the same IP -> Triggers all kind of account/security protections

-psp-

[pspzockerscene]

So after a Teamviewer session we got it figured out though I'm unable to present a 100% explanation:

leroyeur was using Opera and was logged in into his julesjordan.com account.

JDs current User-Agent is a Firefox User-Agent.
I'm using Chrome myself and I was not able to reproduce the issue.
Now I told him to use another browser once so he tested with a Firefox and immediately ran into the "security verification".
Once he solved that, in the Firefox browser, it was working fine in JDownloader - basically it seems like doing this kind of whitelisted the browsers Firefox + Chrome for the current users' IP which alkso whitelisted JD.

My final conclusion is:
- Julesjordan seems to bind this check on IP + Browser-Type
- Seems like they're not making a difference between Chrome and Firefox browsers but they do separate that from Opera
- The security-verification handling in JD either doesn't work properly or they are using some other traits to differentiate between browser/bot --> I wasn't able to reproduce it so I wasn't able to check/fix it

For now this one is solved.
If anyone has more information on thus or can reproduce further issues, please go ahead and report them to us.

-psp-
EDIT

Thanks for the nice chat leroyeur!

[pspzockerscene]

Additional thoughts:
Jiaz reminded me that the Opera browser has an integrated VPN.
I haven't paid attention to this when we had our Teamviewer session.
This could also be part of the issue.

I know that you're using a VPN globally on your system but we haven't checked for the Opera VPN.
If you want you could also check this:
1. Open ipcheck0.jdownloader.org in Opera and write down the result.
2. Open ipcheck0.jdownloader.org in another browser and note down the result.

Different results = Opera VPN is active.
Same results = Opera VPN is not active and thus has not caused this issue.

-psp-