JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 11.08.2022, 19:31
lucia lucia is offline
Fibre Channel User
 
Join Date: Nov 2010
Location: Earth
Posts: 118
Default CrawlJob links sometimes come up as offline.

Hi,

Just fiddling around with some CrawlJob entries.
I have created a CrawlJob with multiple links.

Sample:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
PS: Rename to randomname.crawljob

However some links come up as offline.
Why is that?
When I enter them manually, they are always online.

Ref:
11.08.22 19.52.21 <--> 11.08.22 20.27.38 jdlog://6791035302851/

Last edited by lucia; 11.08.2022 at 19:42.
Reply With Quote
  #2  
Old 11.08.2022, 19:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,943
Default

@lucia: crawljob contains how many links? are they generic http links or links with dedicated JDownloader plugin? what host?
Your log does not contain any error/activity. Is it about instagram links?

the log does not contain any error. how many is *some links*? 10 out of 100 ?
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 11.08.2022 at 19:42.
Reply With Quote
  #3  
Old 12.08.2022, 18:08
lucia lucia is offline
Fibre Channel User
 
Join Date: Nov 2010
Location: Earth
Posts: 118
Default

Hi,

Out of total 41... :D
Which is now 6 twitter and 35 instagram links...
Like 10% is coming up as "No valid account found" and nearly 65% parses about 5~80 images/videos and then comes up the "No valid account found" error...

Was this a parser error? Wow, now I am puzzled with this...

I now updated the link and made it spit out ~41 random.crawljob files to be parsed in order.
But this does not seem to fix my issue...

Strange...
Reply With Quote
  #4  
Old 17.08.2022, 18:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,943
Default

@lucia: log does not contain any error but shows that both (instagram and twitter) are still processing your input
can you please provide a new smaller log? eg restart JDownloader and just add a crawljob with 2 entries or 1 crawljob and wait for issue to show up.
Please know that instagram may block your crawling attempts

also do the linky really come back as offline? I'm asking because log shows
Quote:
Failed to find any crawlable content because of user settings. Crawlable but skipped 2 item(s) due to users' plugin settings.
such entries may get added as dummy retry links so you can modify your settings and retry maybe you mistake those with offline results?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 20.08.2022, 19:12
lucia lucia is offline
Fibre Channel User
 
Join Date: Nov 2010
Location: Earth
Posts: 118
Default

I can provide you with a smaller log of course. Let me see and find 2 that end prematurely. I have created a new IG login cookie.

20.08.22 20.08.06 <--> 20.08.22 20.11.28 jdlog://0439211370661/

1st one done OK.
2nd and 3rd skipped after getting 5 files and saying no valid account is found...
But when I feed them in manually, they are processed with no issues.(in the log, I haven't)
Maby a wait-time after one crawljob file might do the trick?
I am just guessing...
Reply With Quote
  #6  
Old 23.08.2022, 14:54
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,173
Default

Well a logout can happen at any point of time and it is it hard to detect that.
On top of that, most user will be using the cookie login method for Instagram so JD is not able to refresh the cookies on its own this means there is no way for it to continue automatically.

Were you able to add more links without needing to add fresh Instagram cookies?
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 03.09.2022, 08:44
lucia lucia is offline
Fibre Channel User
 
Join Date: Nov 2010
Location: Earth
Posts: 118
Default

But when I add another link, it processes with no issues.
I am using the Cookie login method.
I am using the same account, not freshing up the IG cookies..
Reply With Quote
  #8  
Old 03.09.2022, 16:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,173
Default

I've made some changes to the IG plugin yesterday maybe they've improved the situation for you(?)
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 04.09.2022, 10:46
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,943
Default

Quote:
Originally Posted by pspzockerscene View Post
I've made some changes to the IG plugin yesterday maybe they've improved the situation for you(?)
@lucia: please test and provide feedback
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 21.09.2022, 00:03
lucia lucia is offline
Fibre Channel User
 
Join Date: Nov 2010
Location: Earth
Posts: 118
Default

Parsing part seems to be running okay but the IG somehow keeps disabling my account saying that I might have entered my login info to a site and requests me to re-login after changing my account password and thus update my login cookie. And I have to change the password and verify m phone/email before setting a new pw and getting my new login cookie...
Will test more...
Reply With Quote
  #11  
Old 21.09.2022, 12:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,943
Default

Quote:
Originally Posted by lucia View Post
Parsing part seems to be running okay but the IG somehow keeps disabling my account saying that I might have entered my login info to a site and requests me to re-login after changing my account password and thus update my login cookie. And I have to change the password and verify m phone/email before setting a new pw and getting my new login cookie...
Thanks for the feedback!
this happens when they detect unusual activity on the account which it is when you crawl/mass download stuff That's why you should not use your *real* account for scripts/tools as you might risk loosing the account permanently

you can try to *play* with Settings->Plugins->instagram.com "define global request limit"
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 05:54.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.