[User feedback required] CrawlJob links sometimes come up as offline.

[lucia]

Hi,

Just fiddling around with some CrawlJob entries.
I have created a CrawlJob with multiple links.

Sample:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
PS: Rename to randomname.crawljob

However some links come up as offline.
Why is that?
When I enter them manually, they are always online.

Ref:
11.08.22 19.52.21 <--> 11.08.22 20.27.38 jdlog://6791035302851/

[Jiaz]

@lucia: crawljob contains how many links? are they generic http links or links with dedicated JDownloader plugin? what host?
Your log does not contain any error/activity. Is it about instagram links?

the log does not contain any error. how many is *some links*? 10 out of 100 ?

[lucia]

Hi,

Out of total 41... :D
Which is now 6 twitter and 35 instagram links...
Like 10% is coming up as "No valid account found" and nearly 65% parses about 5~80 images/videos and then comes up the "No valid account found" error...

Was this a parser error? Wow, now I am puzzled with this...

I now updated the link and made it spit out ~41 random.crawljob files to be parsed in order.
But this does not seem to fix my issue...

Strange...

[Jiaz]

@lucia: log does not contain any error but shows that both (instagram and twitter) are still processing your input
can you please provide a new smaller log? eg restart JDownloader and just add a crawljob with 2 entries or 1 crawljob and wait for issue to show up.
Please know that instagram may block your crawling attempts

also do the linky really come back as offline? I'm asking because log shows

Failed to find any crawlable content because of user settings. Crawlable but skipped 2 item(s) due to users' plugin settings.

such entries may get added as dummy retry links so you can modify your settings and retry maybe you mistake those with offline results?

[lucia]

I can provide you with a smaller log of course. Let me see and find 2 that end prematurely. I have created a new IG login cookie.

20.08.22 20.08.06 <--> 20.08.22 20.11.28 jdlog://0439211370661/

1st one done OK.
2nd and 3rd skipped after getting 5 files and saying no valid account is found...
But when I feed them in manually, they are processed with no issues.(in the log, I haven't)
Maby a wait-time after one crawljob file might do the trick?
I am just guessing...

[pspzockerscene]

Well a logout can happen at any point of time and it is it hard to detect that.
On top of that, most user will be using the cookie login method for Instagram so JD is not able to refresh the cookies on its own this means there is no way for it to continue automatically.

Were you able to add more links without needing to add fresh Instagram cookies?

[lucia]

But when I add another link, it processes with no issues.
I am using the Cookie login method.
I am using the same account, not freshing up the IG cookies..

[pspzockerscene]

I've made some changes to the IG plugin yesterday maybe they've improved the situation for you(?)

[Jiaz]

Originally Posted by pspzockerscene

I've made some changes to the IG plugin yesterday maybe they've improved the situation for you(?)

@lucia: please test and provide feedback

[lucia]

Parsing part seems to be running okay but the IG somehow keeps disabling my account saying that I might have entered my login info to a site and requests me to re-login after changing my account password and thus update my login cookie. And I have to change the password and verify m phone/email before setting a new pw and getting my new login cookie...
Will test more...

[Jiaz]

Originally Posted by lucia

Parsing part seems to be running okay but the IG somehow keeps disabling my account saying that I might have entered my login info to a site and requests me to re-login after changing my account password and thus update my login cookie. And I have to change the password and verify m phone/email before setting a new pw and getting my new login cookie...

Thanks for the feedback!
this happens when they detect unusual activity on the account which it is when you crawl/mass download stuff That's why you should not use your *real* account for scripts/tools as you might risk loosing the account permanently

you can try to *play* with Settings->Plugins->instagram.com "define global request limit"

[lucia]

Nearly no more account validations comes up unless I do twice a day download operations.. Thanks for the change...

[Jiaz]

@lucia: maybe create/use multiple accounts for your crawling/downloading activity to spread the workload on multiple accounts and less *risk* of annoying account validations

[pspzockerscene]

Also there are other scripts on github.com specifically for IG downloading. Maybe they're less likely to get your account banned than JD (didn't test.).