JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #21  
Old 17.11.2020, 03:53
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@pspzockerscene

Great job!

Can I ask you what you change to retrieve text from that tag ? What you write ?
Reply With Quote
  #22  
Old 17.11.2020, 09:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,342
Default

@nathan1: he updated the native plugin to fetch the title as you want it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 18.11.2020, 11:29
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@pspzockerscene

Seems that in your last update you disable my last fetching request
Please see LOG

Code:
18.11.20 09.16.21 <--> 18.11.20 09.15.44 jdlog://9615425302851/
Reply With Quote
  #24  
Old 18.11.2020, 14:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,342
Default

@nathan1: was my fault, wait for next update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 18.11.2020, 17:47
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@Jiaz

ok,thanks
Reply With Quote
  #26  
Old 19.11.2020, 16:30
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@Jiaz & psp

I see that plugin have problems to decrypt and fetch <title> names if links are from file.karelia.ru

If you test these urls

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

you return this strange folder names

Code:
923zjt
rvw36q
wn93q9
qgw44f
gv79df


LOG
Code:
19.11.20 14.03.23 <--> 19.11.20 14.31.10 jdlog://9645425302851/
Reply With Quote
  #27  
Old 19.11.2020, 17:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Updated our file.karelia.ru crawler to only set packagenames for folders with more than 1 items.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #28  
Old 21.11.2020, 05:47
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default host url missing from Copy Information (Action)

I think that there are some problems with CopyToClipboard Action via rightclick -> context menu

I add these additional tags in (Action) Copy Information

Code:
{type};{name};{url};{url.container};{url.origin};{packagename};{url.referrer}
but JD2 don't copy also hoster's url. I try to explain: I copy these links

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Inside these pages are links like

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

but all external hoster's url like yadi.sk or sampo.ru are missing from (Action) Copy Information. These information don't appears.
When I copy information







What additional tag do I need? These are insufficient

Code:
{type};{name};{url};{url.container};{url.origin};{packagename};{url.referrer}
LOG
Code:
21.11.20 03.26.47 <--> 21.11.20 03.46.28 jdlog://4695425302851/

Last edited by Jiaz; 21.11.2020 at 13:25.
Reply With Quote
  #29  
Old 21.11.2020, 08:12
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

The data returned by {url} will vary, depending on the url sort order in Settings > User Interface. Use {url.content} instead, to get the download url.
Reply With Quote
  #30  
Old 21.11.2020, 16:56
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

Quote:
Originally Posted by mgpai View Post
The data returned by {url} will vary, depending on the url sort order in Settings > User Interface. Use {url.content} instead, to get the download url.
Thank you !
Reply With Quote
  #31  
Old 21.11.2020, 17:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,342
Default

@mgpai: Thanks for the fast and correct help
__________________
JD-Dev & Server-Admin
Reply With Quote
  #32  
Old 02.09.2022, 02:59
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@psp

I follow your tips from your last post of khinsider to apply for metalarea

Quote:
1. Go to Settings -> Advanced Settings -> Search for LinkCollector.dolinkcheck --> Disable this
2. Now add some- or all links.
You will notice that they will get added with a blue questionmark and without onlinestatus and with unknown filesize.
but seems that rate limit and disabling LinkCollector check is not sufficient. For example I copy about 500 metalarea urls but JDownloader still crawls a limitated number of urls (about 185)
Note: clipboard observer is enabled

LOG
Code:
01.09.22 23.15.02 <--> 01.09.22 23.52.12 jdlog://7369211370661/
I noticed that even if I have disabled the linkcollector check, JD does a check, even if partial, of the links, for example it gives me a high number of offline files.
In theory it shouldn't just show me the package name only (and this is great), but without having to check any links in it?




Why check me anyway if the files are offline even after disabling the linkcollector check? I'm only interested in coming up with the name of the packages, which it does correctly and only later can I check their status online - manually or when I download them.
Reply With Quote
  #33  
Old 02.09.2022, 14:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Quote:
Originally Posted by nathan1 View Post
but seems that rate limit and disabling LinkCollector check is not sufficient.
In the case of metalarea it's not even clear if they have any kind of rate-limit...

Quote:
Originally Posted by nathan1 View Post
For example I copy about 500 metalarea urls but JDownloader still crawls a limitated number of urls (about 185)
Please provide the following information for us to check:
- All of those 500 URLs
- Your metalarea username + password

Quote:
Originally Posted by nathan1 View Post
I noticed that even if I have disabled the linkcollector check, JD does a check, even if partial, of the links, for example it gives me a high number of offline files.
That's because some URLs need to be crawled first as they could return multiple URLs.
This can't be turned off as JD would simply do nothing then
Also some crawlers will do the linkcheck right away because they've accessed the URL already so yes even with disabled linkcheck, some items will be displayed as online/offline with filename/filesize information set.

Quote:
Originally Posted by nathan1 View Post
In theory it shouldn't just show me the package name only (and this is great), but without having to check any links in it?
No.

Quote:
Originally Posted by nathan1 View Post
Why check me anyway if the files are offline even after disabling the linkcollector check? I'm only interested in coming up with the name of the packages, which it does correctly and only later can I check their status online - manually or when I download them.
I'm sorry but without having your testlinks I'm unable to tell what exactly happened but it doesn't look like a bug.
I can imagine that some of your metalarea links go to a 404 error-page which the crawler will detect and display them as offline.
This is not a real "online check" but as explained the crawler needs to access the added link anyways so that is not a bug.

Also, a package in JDownloader cannot be empty it needs to contain at least one item.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #34  
Old 02.09.2022, 15:46
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@psp

ok, the problem is that it does not detect many links: if you copy and paste all 500 links in bulk, JDownloader at some point freezes and "runs idle" for 1-2 minutes and in any case does not scan all the links. Instead, if you copy and paste 10-20 links it scans them and detects all 20.

What I don't understand is why the scan is conditioned by the number of links. Maybe there is some setting or timer to improve?

I sent you credentials + 500 urls metalarea links
Reply With Quote
  #35  
Old 02.09.2022, 16:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Quote:
Originally Posted by nathan1 View Post
ok, the problem is that it does not detect many links: if you copy and paste all 500 links in bulk, JDownloader at some point freezes and "runs idle" for 1-2 minutes and in any case does not scan all the links.
I will take a look at it but as a poweruser you will always run into some kind of limits.
If you're frequently running into rate-limits, consider adding links with a delay of X seconds.
This should be easily possible using external scripts or even an EventScripter script:
https://support.jdownloader.org/Know...event-scripter

Quote:
Originally Posted by nathan1 View Post
What I don't understand is why the scan is conditioned by the number of links. Maybe there is some setting or timer to improve?
I'm sorry but I didn't understand neither of those two sentences.

Quote:
Originally Posted by nathan1 View Post
I sent you credentials + 500 urls metalarea links
I'll check it...
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #36  
Old 02.09.2022, 16:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

It was indeed a rate-limit.
Metalarea will return http error-code 400 when that limit is reached.
For the next update I've added some measures to try to prevent running into the rate-limit.
These measures include:
- Wait 1000ms between requests
- Limit max simultan crawler instances for metalarea to 1
- Return dummy URLs for retrying in case rate-limit is hit

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #37  
Old 02.09.2022, 19:29
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@psp
Thank you for this update
Reply With Quote
  #38  
Old 03.09.2022, 18:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Seems like is not a classic rate limit neither is it a sophisticated one:
They're simply blocking your current session after X (about 250) requests no matter how fast these requests are performed.

I'm still working on it but I think I should be able to add auto handling and remove the requestInterval so the crawler can crawl fullspeed.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #39  
Old 05.09.2022, 13:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Done. Rate-limit for metalarea links shouldn't be a problem anymore after the next set of plugin updates.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #40  
Old 05.09.2022, 22:47
nathan1 nathan1 is offline
JD VIP
 
Join Date: Apr 2012
Posts: 395
Default

@psp

Thank you very much !
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 12:18.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.