JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #41  
Old 03.02.2020, 22:07
wtfiwinomgs
Guest
 
Posts: n/a
Default

i have this problem as well!
Reply With Quote
  #42  
Old 03.02.2020, 22:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by wtfiwinomgs View Post
i have this problem as well!
Please read the previous post then you might be able to find out why ...
https://board.jdownloader.org/showpo...3&postcount=40

I've released an update which might be able to completely fix single images/video support - please try and post your feedback here!
I'll check the profile crawler tomorrow - it remains broken for now.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #43  
Old 07.02.2020, 20:32
Mercyguy's Avatar
Mercyguy Mercyguy is offline
Vacuum Cleaner
 
Join Date: Nov 2009
Posts: 18
Default

Quote:
Originally Posted by pspzockerscene View Post
Nope your content is public thus an account is not required.
Actually it's not true, Twitter have private accounts posts only visible to approved followers. I think it's good to have account log-in in future. But for now, having profile crawler fixed is more important. No rush =)

-Cheers.

Last edited by Mercyguy; 07.02.2020 at 20:36.
Reply With Quote
  #44  
Old 08.02.2020, 09:08
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

It does appear that profile and media scraping is working or on the right track of working.

I would like to report one issue before this goes on further.

The filenaming of downloaded files is a bit off for me. It's not a terrible issue, but it's a change from how Jdownloader previously handled naming of downloads.

So for example, if I scraped and downloaded this twitter post while doing a profile scrape

**External links are only visible to Support Staff****External links are only visible to Support Staff**

The file would normally in the past have come out as 1226027388355526658_EQO5aPkVUAIqQEH

The first set of numbers relating to the numbers in the URL and the last string of letters and digits relating to the file.

I would appreciate if I could have the option to set that naming scheme again as a great coincidence of that naming scheme is that post numbers on Twitter are always in ascending order. So it puts images and videos in chronological order in my folders.

Final note though, it doesn't appear as though this issue affects videos. Videos still get the URL_FileName naming scheme as before, so that may be worth looking into.
Reply With Quote
  #45  
Old 08.02.2020, 09:10
Telephoton
Guest
 
Posts: n/a
Default Twitter is downloading thumbnails instead of videos

Just as the title says. Jdownloader is downloading thumbnails/images/jpg instead of videos
Reply With Quote
  #46  
Old 09.02.2020, 03:39
Telephoton
Guest
 
Posts: n/a
Default

Jdownloader appears to be downloading jpg instead of videos for me. Maybe that is related to the way videos are named as mentioned in the above reply? It seems that images work ok though. I've tried renaming the jpg files to mp4 but that doesn't work because the files that are downloaded are actually jpg files. Apparently it's been doing that for the past few days and all the videos I downloaded are lost.

A post was made a few days or a week ago about videos not working, after which it seemed that the video downloading worked again but I guess it was just downloading jpg files the entire time.
Reply With Quote
  #47  
Old 10.02.2020, 10:06
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

In the interest of double checking your claim (cause I thought I was downloading videos just fine on my end) I have found that I am also suffering the same issue.

For testing purposes, I'm using this profile to test

**External links are only visible to Support Staff****External links are only visible to Support Staff**

10.02.20 02.55.45 <--> 10.02.20 03.06.28 jdlog://2925430900751/

And my logs
Reply With Quote
  #48  
Old 10.02.2020, 18:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by Mercyguy View Post
Actually it's not true, Twitter have private accounts posts only visible to approved followers.
Account support did get Twitter accounts banned before which was of course not our intention.

If you have private twitter URLs and an example account, PN me.


Merged twitter threads!

@Telephoton & Akasen
Fixed!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #49  
Old 16.02.2020, 09:06
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default Twitter Download Naming Scheme Changed

So I'm just gonna make a formal post about this odd issue.

Simply put, for the longest time Jdowloader would download twitter videos and images by a scheme of [Tweet-ID]_[Image-ID] or whatever you want to call it.

Basically, a download would come as "608048483195617280_CHA431wWcAAhp7l". Now it's just "CHA431wWcAAhp7l"

What I want is for the previous notation back, or at least the option for it. It was very helpful in organizing images by date since all tweets posted are in ascending order.
Reply With Quote
  #50  
Old 17.02.2020, 22:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Merged Twitter threads.

Hi Akasen,
please add example URLs and describe exactly (e.g. with real examples) how you'd like the filenames to look like.

The filenames have not been changed intentionally - this was the side-effect of the recent twitter website changes.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #51  
Old 18.02.2020, 10:25
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Certainly.

So for example, if this post was downloaded (**External links are only visible to Support Staff****External links are only visible to Support Staff** )

what I would like is for the file to be named "1197122159883063296_EJ0IPocUUAEhz9M" and "1197122159883063296_EJ0IPofU8AANM3". This was how it was previously for some time now.

With regards to videos and anything downloaded as an MP4, the file would simply be just the URL twice over. So something like "921155373578506241_921155373578506241" **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #52  
Old 18.02.2020, 16:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

I've updated the filenames accordingly for the next update

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #53  
Old 19.02.2020, 07:27
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Much appreciated!
Reply With Quote
  #54  
Old 22.02.2020, 09:43
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Hi! I have one other issue now.

Previously Jdownloader could take the specific link for a Twitter's Media tab and scrape ONLY the media. Now it doesn't seem to do just that anymore and scrapes the whole Twitter feed regardless. Any chance this could be looked into?

Twitter link for example
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #55  
Old 24.02.2020, 16:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

What do you mean by "only" the media?
It grabs pictures and videos at the moment - how do you define "media"? What gets crawled that you don't wand?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #56  
Old 25.02.2020, 06:59
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Twitter has a tab for only media, that is the media a user has uploaded.

When you put in the URL the form of "**External links are only visible to Support Staff**, Jdownloader use to only scrape just THAT section of the twitter feed, thus limiting the scraping to the original content and uploads that user uploaded.

Last edited by Akasen; 25.02.2020 at 09:46.
Reply With Quote
  #57  
Old 25.02.2020, 15:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

You are indeed right.

Try again after the next update.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #58  
Old 26.02.2020, 14:12
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

So just to double check something here and make sure I'm not complaining about an old issue that we've gone over is difficult to fix, and also that I'm not complaining about a problem when a plugin hasn't updated yet, I have a problem to throw out there again.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

26.02.20 06.58.01 <--> 26.02.20 07.04.10 jdlog://0080715302851/

Two sets of links and a log that should have my scraping and downloading of them. The first link there shows that the user uploaded media should be 860 photos/videos, but only picking up on 187. Second ones weird and Twitter says 93, but Jdownloader picked up 118, I just assume that Twitter counts single posts as a "photo/video post" and not the actual number of images contained. (EDIT: Okay, that's actually right. I'm over thinking that second ones numbers)

Anyways, my question is this: with the first one am I simply just experiencing that issue where Jdownloader can only hit the Twitter API so many times on one page? Or was that potentially fixed before and broken again?

Last edited by Akasen; 26.02.2020 at 14:15.
Reply With Quote
  #59  
Old 26.02.2020, 18:13
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

@Akasen
Indeed media handling was still not working property!
I've fixed it with the next update and added additional loggers so I can easily see "how far" the crawl process is.
Your 2nd URL got me a total of 1117 items.


Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #60  
Old 27.02.2020, 05:17
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Quote:
Originally Posted by pspzockerscene View Post
@Akasen
Indeed media handling was still not working property!
I've fixed it with the next update and added additional loggers so I can easily see "how far" the crawl process is.
Your 2nd URL got me a total of 1117 items.


Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!


-psp-
Woah, weird, that's a totally different result than on my end, huh.
Reply With Quote
  #61  
Old 27.02.2020, 16:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by Akasen View Post
Woah, weird, that's a totally different result than on my end, huh.
Sorry my bad, I ment the first URL - this was just to show you that it will work after the update.

Additionally I have added a plugin setting for the twitter plugin which is on by default:
It will by default always "only" grab the "/media" as this is what users will usually want and it will by far not take as long as crawling every status of a profile with retweets.
In your example it was like 800 media posts and 16 000 posts total.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #62  
Old 27.02.2020, 18:18
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Quote:
Originally Posted by pspzockerscene View Post
Sorry my bad, I ment the first URL - this was just to show you that it will work after the update.

Additionally I have added a plugin setting for the twitter plugin which is on by default:
It will by default always "only" grab the "/media" as this is what users will usually want and it will by far not take as long as crawling every status of a profile with retweets.
In your example it was like 800 media posts and 16 000 posts total.

-psp-
Oh that's a wonderful addition! Thank you so much for that!
Reply With Quote
  #63  
Old 28.02.2020, 01:58
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

I'd like to report in that so far as I can tell, everything appears to be working just fine for the time being. I'm running a couple scrapes on on accounts I'd had issues with in the past and other stuff to test things out, but "so far so good"

EDIT:

Wow, this is actually working exceptionally well so far.

Especially the login information. I had read some issues occurring before that I feared might make me be unable to use my twitter account for downloads. It's good to see that's also working too.

Fantastic job!

Last edited by Akasen; 28.02.2020 at 03:01. Reason: Further appreciation and thanks
Reply With Quote
  #64  
Old 02.03.2020, 17:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Thanks for the feedback.

I was not able to fix the login properly but my included cookie-workaround did work quite well.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #65  
Old 24.03.2020, 22:06
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

So small update on this front, video and gif downloading (which I think is just the same thing really on Twitter's end) is working weird.

What will happen is that I'll scrape the media of some account and the video and gifs don't download sometimes. I think it's been happening before, but I've only sort of just noticed now.

It's not like it's impossible to get Jdownloader to finally download them, but I'm also unsure as to what's making it do so. It's either that constantly doing a "check online status" on the link and a "force start" are somehow doing it, or doing a mass force start on a bunch of failed video downloads causes most of them to get a hit.

I've included a log that should help and a link to a post that was failing and also a direct link to the gif in the post if that at all somehow helps give a better idea on how to improve checks for this kind of media since video media is usually heavily nested under multiple tags.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

24.03.20 02.25.57 <--> 24.03.20 15.58.59 jdlog://8868715302851/
Reply With Quote
  #66  
Old 25.03.2020, 15:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

I've fixed the errorhandling as it displayed "File not found" in this case although an error with retry was supposed to happen.

This will not necessarily improve the situation you'll have to test it.

Your log contains this failure over 700 times.
Question is why it happens - maybe we're doing too many requests in a short time but according to your log we're using a wrong token which is not true.

What you could try is:
Add your twitter account to JDownloader using the following workaround:
1. Get the free- and open source Browser addon "EditThisCookie".
2. Export your twitter cookies
3. Add a new Twitter account in JDownloader. Enter your username in the corresponding field and enter the exported cookies in the password field.
4. Try downloading

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #67  
Old 25.03.2020, 21:00
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Yeah, I'll give that a whirl best I can just a couple questions before I proceed to any testing.

So I'm getting a message stating that the twitter account plugin is outdated when I change the password. This expected behavior at the moment?

Also just to make sure of something with cookies, I'd be right to use the auth_token value as the password, right?

With those questions out of the way though, I do have some results to give at the very least.

I was able to get a number of videos and gifs to download without issue. I assume given what's been said so far, it does appear my account details being inputted were to blame, though curious it only affected videos and gifs.

However though, I did run into an issue while downloading many gifs that I had a host problem error come up

25.03.20 14.32.38 <--> 25.03.20 14.45.52 jdlog://4898715302851/

Following that though, I decided to try and see if this would persist with any other twitter account, and to my surprise, Jdownloader was refusing to even crawl
25.03.20 14.49.32 <--> 25.03.20 14.51.46 jdlog://7898715302851/

Actually, PS, I actually remedied this previous issue by simply using a container that had no cookies relating to login information and trying to view a couple other web pages. The pages I was trying to scrape were not letting me view them at all, but the moment I checked someone elses twitter and came back, the feed was loading.
Reply With Quote
  #68  
Old 26.03.2020, 12:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by Akasen View Post
Yeah, I'll give that a whirl best I can just a couple questions before I proceed to any testing.

So I'm getting a message stating that the twitter account plugin is outdated when I change the password. This expected behavior at the moment?
Well basically our twitter login process is broken due to recent twitter website changes but there is a workaround (the already mentioned cookie thing) for which I will post instructions below.

Quote:
Originally Posted by Akasen View Post
Also just to make sure of something with cookies, I'd be right to use the auth_token value as the password, right?
No you're on the wrong path here ...

Quote:
Originally Posted by Akasen View Post
With those questions out of the way though, I do have some results to give at the very least.

I was able to get a number of videos and gifs to download without issue. I assume given what's been said so far, it does appear my account details being inputted were to blame, though curious it only affected videos and gifs.

However though, I did run into an issue while downloading many gifs that I had a host problem error come up

25.03.20 14.32.38 <--> 25.03.20 14.45.52 jdlog://4898715302851/

Following that though, I decided to try and see if this would persist with any other twitter account, and to my surprise, Jdownloader was refusing to even crawl
25.03.20 14.49.32 <--> 25.03.20 14.51.46 jdlog://7898715302851/
I've added more errorhandling - according to your recent logs you were rate-limited by twitter which is to be expected when doing a lot of requests in a short time.
This can even happen via browser / if it happens in JD, you might be rate-limited in browser too and are not able to load/see twitter content for some time.

I've added a wait-time for this case.


Quote:
Originally Posted by Akasen View Post
Actually, PS, I actually remedied this previous issue by simply using a container that had no cookies relating to login information and trying to view a couple other web pages. The pages I was trying to scrape were not letting me view them at all, but the moment I checked someone elses twitter and came back, the feed was loading.
Sorry I don't understand that.

Please also keep in mind:
- Read the previously linked Update FAQ
- Short answer for lazy users: Changes are NOT (NOT, NOT, NOT!) always released immediately after ourstaff answers in the forum --> My recently mentioned twitter improvements are NOT yet released!

I will post the Cookie-Login instructions in my next post!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #69  
Old 26.03.2020, 13:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Instructions on how to login into twitter accounts using the "workaround cookie login" method.

Warning: This is only a proof-of-concept workaround which may work or may not and the process of this cookie login may also get changed- or removed in the future!

Disclaimer: JDownloader may perform a lot of http requests which could potentially get your twitter account banned!

1. Get the free- and open source addon "EditThisCookie" for your browser: editthiscookie.com.
2. Open the twitter.com main page and log in into your twitter account.
3. While your twitter.com tab is still open, Click on the "EditThisCookie" icon in the upper right corner of your browser window --> Export your cookies
Screenshot: Click on the "SPOILER/show" to see the screenshot
Spoiler:

4. Now you should have a really long string in your clipboard looking like this but much longer:
Code:
[
{
    "domain": ".twitter.com",
    "hostOnly": false,
    "httpOnly": true,
    "name": "_twitter_sess",
    "path": "/",
    "sameSite": "unspecified",
    "secure": true,
    "session": true,
    "storeId": "0",
    "value": "CENSORED",
    "id": 1
},[...]
Go in JDownloader --> Account Manager --> Add account --> Twitter.com --> Enter your username in username and enter that long String as your password --> Confirm that --> If everything is working as expected, your twitter account should get accepted by JDownloader which means you can e.g. add URLs which lead to restricted content e.g. for which an account is required.

I recommend you to only enable your twitter account to download content for which an account is required!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #70  
Old 27.03.2020, 03:44
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

Yeah, regarding that last statement of mine, really difficult to explain and even running it in my head makes no sense and isn't helpful.

As for Twitter logins, I will report that it's working again, however I didn't need to input the string of cookies. Username and Password input is working just fine on my end so far as I can tell. Not sure if that's expected on your end, or if I'm lucky.

I will say though that I have inputted the long string of cookies into the password and it's working too just fine. So good to know how to do that.
Reply With Quote
  #71  
Old 27.03.2020, 14:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Thanks for your feedback

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #72  
Old 31.03.2020, 09:09
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

So I absolutely hate to bring up another issue so soon, but I seem to have a problem with the scraper, and it's making me scratch my head a bit so my explanation is likely way off from what I'm actually looking at.

So what I think I'm experiencing is the scraper missing tweets or something.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

With this one here, I had done a scrape and found that it only gets 1333 hits of some 2748 tweets (at time of typing), and I know that's not right cause it should at least be more than that since when twitter reports "[number] Photos & Videos", it's only reporting individual tweets to that metric and not the contents. I checked to see if it was only going so far back, but it at the very least picked up the first image ever posted on the account.
____________________
NSFW warning
**External links are only visible to Support Staff****External links are only visible to Support Staff**

This one was honestly the one that tipped me off, but made me think it was something different. See, the scraper only seemed to scrape til some point in 2019, and this accounts tweets go as far back as 2016. I've tried to double check with this one for that to be the case and this does seem to be the case that it's only going so far back. and not skipping anything

Like I know at this point twitter is becoming a bit of a chore, but I would appreciate any insight and assistance on this.

31.03.20 01.54.05 <--> 31.03.20 02.02.06 jdlog://5690815302851/
Reply With Quote
  #73  
Old 31.03.2020, 16:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Hi again Akasen,

I've investigated this.

It seems like twitter has a max. number of tweets you can see/go back.
Code:
641|twitter.com_jd.plugins.decrypter.TwitterCom 31.03.20 15:36:59 - INFO [ jd.plugins.decrypter.TwitterCom(crawlUserViaAPI) ] -> Numberof tweets on current page: 0 of expected max 20
641|twitter.com_jd.plugins.decrypter.TwitterCom 31.03.20 15:37:01 - INFO [ jd.plugins.decrypter.TwitterCom(crawlUserViaAPI) ] -> Numberof total tweets crawled: 829 of expected total 2748
By default, twitter will return 20 tweets per page --> Every tweet may contain a different amount of downloadable media.

Basically we noticed this in the past as well.
You can even check this via browser by going back as far as 41 "pages" which means 40x reloading by scrolling down.

The issue is also that a lot of websites would allow to e.g. "start at position 500".
Twitter however uses so called "cursors" which means to access the next page, you will have to get a token which is only available on the previous page so even if I wanted, I would not be able to give you any options to e.g. start at position 800 in this case.

For your other URL, it finds 200 objects which should be all as it only contains 162 tweets.

I could now e.g. experiment and display more objects per page but although thiy may return some more objects, I we would run into similar issues with URLs containing even more objects.

I recommend you to:
- Test via browser and see how far you can get and if you can e.g. get more than JD does
- Search the Internet for other Twitter downloader tools --> If you find one that does a better job than our crawler, let me know and I'll look into it again

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #74  
Old 31.03.2020, 21:45
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

I'll look into other options, though there's one other program I use for other purposes

**External links are only visible to Support Staff****External links are only visible to Support Staff**

They have a fair share of open and closed issues relating to twitter. But, it seems they're running into
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

However, with the first link I gave, it still only got 1333 items from its scrape. HOwever the other link I gave, instead of the 98 items, it actually did download the full 200 items.

I've also given Jdownloader another spin just now though to see if it too could find the objects, it did. The first link returned four more objects than before, but that's on one hand because the account uploaded four more images.

I'm providing a log just in case, I find it both interesting and frustrating that **External links are only visible to Support Staff****External links are only visible to Support Staff** gave me one result in the last seven or so hours, and I wake up to find it giving a different result, even if it's the one I'd have wanted

31.03.20 14.22.59 <--> 31.03.20 14.30.12 jdlog://8711815302851/
Reply With Quote
  #75  
Old 01.04.2020, 12:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Hm as said, I get 200 items when I add that one of your two URLs.
Now I even get 201.

According to the github tickets of this other software, the twitter API is kinda random.

Unfortunately I do not have the time to do big experiments on it and it seems like it is working fine for most of all of our current users so I do not want to add experimental code.

According to the tickets, changing the "filter" values and also the User-Agent may bring more results.

We are open source so if you want you can grab our code and play around with it:
https://jdownloader.org/knowledge/wi...nt/get-started

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #76  
Old 02.04.2020, 00:34
Akasen Akasen is offline
DSL Light User
 
Join Date: Jul 2017
Posts: 34
Default

I can attempt to try my hand at potentially contributing, but I will admit that my skills as a programmer have been lacking. I'll definitely take a look into it though and poke around, but I'm afraid if something breaks again, I'm probably coming back in here with that problem. And maybe some rambling about the source code, lol
Reply With Quote
  #77  
Old 02.04.2020, 11:39
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

I treid to comment the code well so you might be able to change stuff even without much knowledge.

Please keep in mind that other tools may use completely different Twitter APIs - I'm using the "website API" for our plugin.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 23:57.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.