[H-Visage]

Hi there, this may have been asked before, but I noticed that when I attempt to scrap 20 links, they aren't always crawled? So I think I'm getting limited? Has there been a fix for this? I tried readjusting my rates but it still happens.

[pspzockerscene]

There won't ever be a "fix" for this.
Setting higher request-limits is the only thing you can do.

Are you maybe using a VPN or a shared network IP such as a university/company IP?
Such IPs tend to get rate-limited more often.

[H-Visage]

No VPN, just my home PC. Do you have a recommended threshold rate limit to set for twitter?
Also, does crawling Texts count towards the limit? If I disable that would that help? Cause all I care about is media.

[pspzockerscene]

Originally Posted by H-Visage

Do you have a recommended threshold rate limit to set for twitter?

No. Trial and error.

Originally Posted by H-Visage

Also, does crawling Texts count towards the limit?

No.

[H-Visage]

Alright no problem. Appreciate the reply

[az0]

@pspzockerscene Hi what do you think about limits

Spoiler:

[pspzockerscene]

I'am not using the Twitter plugin myself so no idea.
3 seconds delay in between requests should be more than enough but then again it depends on how many items you are adding or downloading.
If you are adding a million items you will be hammering Twitters API for several weeks so even with that delay they still may ban your IP or rate-limit you.

[az0]

Originally Posted by pspzockerscene

I'am not using the Twitter plugin myself so no idea.
3 seconds delay in between requests should be more than enough but then again it depends on how many items you are adding or downloading.
If you are adding a million items you will be hammering Twitters API for several weeks so even with that delay they still may ban your IP or rate-limit you.

Is it possible to add a crawler: action on rate limit reached abort crawl process
to To Twitter plugins

[pspzockerscene]

You mean a setting?
Why?
The plugin informs you about the rate-limit and you can manually abort the crawl process then.

[az0]

You mean a setting?
Why?

the reason
To prevent Twitter from disabling the account or suspending the IP
The addition you requested is like this
The crawler stops when the rate limit is reached
Like this one on plugin instagram

Spoiler:

The plugin informs you about the rate-limit and you can manually abort the crawl process then.

I didn't understand this question exactly please explain

Greetings to you

[pspzockerscene]

Originally Posted by az0

To prevent Twitter from disabling the account or suspending the IP

If you are afraid of that you shouldn't use JDownloader or you should only use Twitter accounts you don't care about.

In fact I just checked in the Twitter plugin code andindeed there is no retry handling when a rate-limit is reached.

Originally Posted by az0

I didn't understand this question exactly please explain

That was not a question.