JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #61  
Old 27.02.2020, 15:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Quote:
Originally Posted by Akasen View Post
Woah, weird, that's a totally different result than on my end, huh.
Sorry my bad, I ment the first URL - this was just to show you that it will work after the update.

Additionally I have added a plugin setting for the twitter plugin which is on by default:
It will by default always "only" grab the "/media" as this is what users will usually want and it will by far not take as long as crawling every status of a profile with retweets.
In your example it was like 800 media posts and 16 000 posts total.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #62  
Old 27.02.2020, 17:18
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Quote:
Originally Posted by pspzockerscene View Post
Sorry my bad, I ment the first URL - this was just to show you that it will work after the update.

Additionally I have added a plugin setting for the twitter plugin which is on by default:
It will by default always "only" grab the "/media" as this is what users will usually want and it will by far not take as long as crawling every status of a profile with retweets.
In your example it was like 800 media posts and 16 000 posts total.

-psp-
Oh that's a wonderful addition! Thank you so much for that!
Reply With Quote
  #63  
Old 28.02.2020, 00:58
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

I'd like to report in that so far as I can tell, everything appears to be working just fine for the time being. I'm running a couple scrapes on on accounts I'd had issues with in the past and other stuff to test things out, but "so far so good"

EDIT:

Wow, this is actually working exceptionally well so far.

Especially the login information. I had read some issues occurring before that I feared might make me be unable to use my twitter account for downloads. It's good to see that's also working too.

Fantastic job!

Last edited by Akasen; 28.02.2020 at 02:01. Reason: Further appreciation and thanks
Reply With Quote
  #64  
Old 02.03.2020, 16:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Thanks for the feedback.

I was not able to fix the login properly but my included cookie-workaround did work quite well.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #65  
Old 24.03.2020, 21:06
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

So small update on this front, video and gif downloading (which I think is just the same thing really on Twitter's end) is working weird.

What will happen is that I'll scrape the media of some account and the video and gifs don't download sometimes. I think it's been happening before, but I've only sort of just noticed now.

It's not like it's impossible to get Jdownloader to finally download them, but I'm also unsure as to what's making it do so. It's either that constantly doing a "check online status" on the link and a "force start" are somehow doing it, or doing a mass force start on a bunch of failed video downloads causes most of them to get a hit.

I've included a log that should help and a link to a post that was failing and also a direct link to the gif in the post if that at all somehow helps give a better idea on how to improve checks for this kind of media since video media is usually heavily nested under multiple tags.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

24.03.20 02.25.57 <--> 24.03.20 15.58.59 jdlog://8868715302851/
Reply With Quote
  #66  
Old 25.03.2020, 14:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

I've fixed the errorhandling as it displayed "File not found" in this case although an error with retry was supposed to happen.

This will not necessarily improve the situation you'll have to test it.

Your log contains this failure over 700 times.
Question is why it happens - maybe we're doing too many requests in a short time but according to your log we're using a wrong token which is not true.

What you could try is:
Add your twitter account to JDownloader using the following workaround:
1. Get the free- and open source Browser addon "EditThisCookie".
2. Export your twitter cookies
3. Add a new Twitter account in JDownloader. Enter your username in the corresponding field and enter the exported cookies in the password field.
4. Try downloading

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #67  
Old 25.03.2020, 20:00
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Yeah, I'll give that a whirl best I can just a couple questions before I proceed to any testing.

So I'm getting a message stating that the twitter account plugin is outdated when I change the password. This expected behavior at the moment?

Also just to make sure of something with cookies, I'd be right to use the auth_token value as the password, right?

With those questions out of the way though, I do have some results to give at the very least.

I was able to get a number of videos and gifs to download without issue. I assume given what's been said so far, it does appear my account details being inputted were to blame, though curious it only affected videos and gifs.

However though, I did run into an issue while downloading many gifs that I had a host problem error come up

25.03.20 14.32.38 <--> 25.03.20 14.45.52 jdlog://4898715302851/

Following that though, I decided to try and see if this would persist with any other twitter account, and to my surprise, Jdownloader was refusing to even crawl
25.03.20 14.49.32 <--> 25.03.20 14.51.46 jdlog://7898715302851/

Actually, PS, I actually remedied this previous issue by simply using a container that had no cookies relating to login information and trying to view a couple other web pages. The pages I was trying to scrape were not letting me view them at all, but the moment I checked someone elses twitter and came back, the feed was loading.
Reply With Quote
  #68  
Old 26.03.2020, 11:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Quote:
Originally Posted by Akasen View Post
Yeah, I'll give that a whirl best I can just a couple questions before I proceed to any testing.

So I'm getting a message stating that the twitter account plugin is outdated when I change the password. This expected behavior at the moment?
Well basically our twitter login process is broken due to recent twitter website changes but there is a workaround (the already mentioned cookie thing) for which I will post instructions below.

Quote:
Originally Posted by Akasen View Post
Also just to make sure of something with cookies, I'd be right to use the auth_token value as the password, right?
No you're on the wrong path here ...

Quote:
Originally Posted by Akasen View Post
With those questions out of the way though, I do have some results to give at the very least.

I was able to get a number of videos and gifs to download without issue. I assume given what's been said so far, it does appear my account details being inputted were to blame, though curious it only affected videos and gifs.

However though, I did run into an issue while downloading many gifs that I had a host problem error come up

25.03.20 14.32.38 <--> 25.03.20 14.45.52 jdlog://4898715302851/

Following that though, I decided to try and see if this would persist with any other twitter account, and to my surprise, Jdownloader was refusing to even crawl
25.03.20 14.49.32 <--> 25.03.20 14.51.46 jdlog://7898715302851/
I've added more errorhandling - according to your recent logs you were rate-limited by twitter which is to be expected when doing a lot of requests in a short time.
This can even happen via browser / if it happens in JD, you might be rate-limited in browser too and are not able to load/see twitter content for some time.

I've added a wait-time for this case.


Quote:
Originally Posted by Akasen View Post
Actually, PS, I actually remedied this previous issue by simply using a container that had no cookies relating to login information and trying to view a couple other web pages. The pages I was trying to scrape were not letting me view them at all, but the moment I checked someone elses twitter and came back, the feed was loading.
Sorry I don't understand that.

Please also keep in mind:
- Read the previously linked Update FAQ
- Short answer for lazy users: Changes are NOT (NOT, NOT, NOT!) always released immediately after ourstaff answers in the forum --> My recently mentioned twitter improvements are NOT yet released!

I will post the Cookie-Login instructions in my next post!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #69  
Old 26.03.2020, 12:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Instructions on how to login into twitter accounts using the "workaround cookie login" method.

Warning: This is only a proof-of-concept workaround which may work or may not and the process of this cookie login may also get changed- or removed in the future!

Disclaimer: JDownloader may perform a lot of http requests which could potentially get your twitter account banned!

1. Get the free- and open source addon "EditThisCookie" for your browser: editthiscookie.com.
2. Open the twitter.com main page and log in into your twitter account.
3. While your twitter.com tab is still open, Click on the "EditThisCookie" icon in the upper right corner of your browser window --> Export your cookies
Screenshot: Click on the "SPOILER/show" to see the screenshot
Spoiler:

4. Now you should have a really long string in your clipboard looking like this but much longer:
Code:
[
{
    "domain": ".twitter.com",
    "hostOnly": false,
    "httpOnly": true,
    "name": "_twitter_sess",
    "path": "/",
    "sameSite": "unspecified",
    "secure": true,
    "session": true,
    "storeId": "0",
    "value": "CENSORED",
    "id": 1
},[...]
Go in JDownloader --> Account Manager --> Add account --> Twitter.com --> Enter your username in username and enter that long String as your password --> Confirm that --> If everything is working as expected, your twitter account should get accepted by JDownloader which means you can e.g. add URLs which lead to restricted content e.g. for which an account is required.

I recommend you to only enable your twitter account to download content for which an account is required!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #70  
Old 27.03.2020, 02:44
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Yeah, regarding that last statement of mine, really difficult to explain and even running it in my head makes no sense and isn't helpful.

As for Twitter logins, I will report that it's working again, however I didn't need to input the string of cookies. Username and Password input is working just fine on my end so far as I can tell. Not sure if that's expected on your end, or if I'm lucky.

I will say though that I have inputted the long string of cookies into the password and it's working too just fine. So good to know how to do that.
Reply With Quote
  #71  
Old 27.03.2020, 13:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Thanks for your feedback

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #72  
Old 31.03.2020, 08:09
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

So I absolutely hate to bring up another issue so soon, but I seem to have a problem with the scraper, and it's making me scratch my head a bit so my explanation is likely way off from what I'm actually looking at.

So what I think I'm experiencing is the scraper missing tweets or something.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

With this one here, I had done a scrape and found that it only gets 1333 hits of some 2748 tweets (at time of typing), and I know that's not right cause it should at least be more than that since when twitter reports "[number] Photos & Videos", it's only reporting individual tweets to that metric and not the contents. I checked to see if it was only going so far back, but it at the very least picked up the first image ever posted on the account.
____________________
NSFW warning
**External links are only visible to Support Staff****External links are only visible to Support Staff**

This one was honestly the one that tipped me off, but made me think it was something different. See, the scraper only seemed to scrape til some point in 2019, and this accounts tweets go as far back as 2016. I've tried to double check with this one for that to be the case and this does seem to be the case that it's only going so far back. and not skipping anything

Like I know at this point twitter is becoming a bit of a chore, but I would appreciate any insight and assistance on this.

31.03.20 01.54.05 <--> 31.03.20 02.02.06 jdlog://5690815302851/
Reply With Quote
  #73  
Old 31.03.2020, 15:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Hi again Akasen,

I've investigated this.

It seems like twitter has a max. number of tweets you can see/go back.
Code:
641|twitter.com_jd.plugins.decrypter.TwitterCom 31.03.20 15:36:59 - INFO [ jd.plugins.decrypter.TwitterCom(crawlUserViaAPI) ] -> Numberof tweets on current page: 0 of expected max 20
641|twitter.com_jd.plugins.decrypter.TwitterCom 31.03.20 15:37:01 - INFO [ jd.plugins.decrypter.TwitterCom(crawlUserViaAPI) ] -> Numberof total tweets crawled: 829 of expected total 2748
By default, twitter will return 20 tweets per page --> Every tweet may contain a different amount of downloadable media.

Basically we noticed this in the past as well.
You can even check this via browser by going back as far as 41 "pages" which means 40x reloading by scrolling down.

The issue is also that a lot of websites would allow to e.g. "start at position 500".
Twitter however uses so called "cursors" which means to access the next page, you will have to get a token which is only available on the previous page so even if I wanted, I would not be able to give you any options to e.g. start at position 800 in this case.

For your other URL, it finds 200 objects which should be all as it only contains 162 tweets.

I could now e.g. experiment and display more objects per page but although thiy may return some more objects, I we would run into similar issues with URLs containing even more objects.

I recommend you to:
- Test via browser and see how far you can get and if you can e.g. get more than JD does
- Search the Internet for other Twitter downloader tools --> If you find one that does a better job than our crawler, let me know and I'll look into it again

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #74  
Old 31.03.2020, 20:45
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

I'll look into other options, though there's one other program I use for other purposes

**External links are only visible to Support Staff****External links are only visible to Support Staff**

They have a fair share of open and closed issues relating to twitter. But, it seems they're running into
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

However, with the first link I gave, it still only got 1333 items from its scrape. HOwever the other link I gave, instead of the 98 items, it actually did download the full 200 items.

I've also given Jdownloader another spin just now though to see if it too could find the objects, it did. The first link returned four more objects than before, but that's on one hand because the account uploaded four more images.

I'm providing a log just in case, I find it both interesting and frustrating that **External links are only visible to Support Staff****External links are only visible to Support Staff** gave me one result in the last seven or so hours, and I wake up to find it giving a different result, even if it's the one I'd have wanted

31.03.20 14.22.59 <--> 31.03.20 14.30.12 jdlog://8711815302851/
Reply With Quote
  #75  
Old 01.04.2020, 11:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Hm as said, I get 200 items when I add that one of your two URLs.
Now I even get 201.

According to the github tickets of this other software, the twitter API is kinda random.

Unfortunately I do not have the time to do big experiments on it and it seems like it is working fine for most of all of our current users so I do not want to add experimental code.

According to the tickets, changing the "filter" values and also the User-Agent may bring more results.

We are open source so if you want you can grab our code and play around with it:
https://jdownloader.org/knowledge/wi...nt/get-started

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #76  
Old 01.04.2020, 23:34
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

I can attempt to try my hand at potentially contributing, but I will admit that my skills as a programmer have been lacking. I'll definitely take a look into it though and poke around, but I'm afraid if something breaks again, I'm probably coming back in here with that problem. And maybe some rambling about the source code, lol
Reply With Quote
  #77  
Old 02.04.2020, 10:39
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

I treid to comment the code well so you might be able to change stuff even without much knowledge.

Please keep in mind that other tools may use completely different Twitter APIs - I'm using the "website API" for our plugin.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 06:07.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.