JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 29.12.2019, 17:08
doiulou doiulou is offline
Modem User
 
Join Date: Dec 2019
Posts: 1
Default Twitter - Scrape date/time limit?

Sorry if I am posting this in the wrong section or am using JDownloader wrong.

I am trying to scrape a twitter profile (all it's media) but somehow with JDownloader I can not get any futher than 19 March 2019? Everything before March 2019 is not downloaded. How can I download these ?

Kind regards,

dioulou
Reply With Quote
  #2  
Old 13.01.2020, 13:43
Akasen Akasen is offline
Junior Loader
 
Join Date: Jul 2017
Posts: 10
Default Twitter - Linkgrabber not doing complete scrape

So I've noticed something recently that's absolutely troubling and I don't know if this has been noted.

While I'm painfully aware of another issue plaguing the Twitter side of things, I feel that at the very I should do my part in at least reporting these things when noticed.

What I'm finding is that when the links are being grabbed, the linkgrabber only goes so far in the past of uploaded files. It's honestly not easy to pin how far Jdownloader goes, or why, but it just does.

So for example on my end, downloading the media of this artist is NEARLY complete, but somehow the last three images (not counting the youtube video posted) just don't get picked up by the Linkgrabber

**External links are only visible to Support Staff****External links are only visible to Support Staff**

And a log
13.01.20 06.17.42 <--> 13.01.20 06.43.06 jdlog://9377330900751/

This test can be repeated across other twitter accounts as well.
Reply With Quote
  #3  
Old 14.01.2020, 01:26
Akasen Akasen is offline
Junior Loader
 
Join Date: Jul 2017
Posts: 10
Default

Funny you're posting this, cause I noticed a similar thing as well. In my case, I was also noticing with someone who hadn't posted since 2017, I wasn't able to get all the stuff from them.

https://board.jdownloader.org/showthread.php?t=82777
Reply With Quote
  #4  
Old 20.01.2020, 21:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 49,888
Default

Twitter web-API seems to have a limit which you should even be able to see via browser.
An old note of mine in our code will also tell this:
Code:
            /* 2016-11-30: Seems like twitter limits their website to a max "load more" calls of 40. */
40 would mean 40 times loading content by scrolling down.

As said - you can simply test this via browser.

I'am not doing this right now as I don't have the time for that (today).

-psp-
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 02:05.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.