JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #41  
Old 03.02.2020, 21:07
wtfiwinomgs wtfiwinomgs is offline
Baby Loader
 
Join Date: Jul 2019
Posts: 9
Default

i have this problem as well!
Reply With Quote
  #42  
Old 03.02.2020, 21:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,356
Default

Quote:
Originally Posted by wtfiwinomgs View Post
i have this problem as well!
Please read the previous post then you might be able to find out why ...
https://board.jdownloader.org/showpo...3&postcount=40

I've released an update which might be able to completely fix single images/video support - please try and post your feedback here!
I'll check the profile crawler tomorrow - it remains broken for now.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #43  
Old 07.02.2020, 19:32
Mercyguy's Avatar
Mercyguy Mercyguy is offline
Vacuum Cleaner
 
Join Date: Nov 2009
Posts: 18
Default

Quote:
Originally Posted by pspzockerscene View Post
Nope your content is public thus an account is not required.
Actually it's not true, Twitter have private accounts posts only visible to approved followers. I think it's good to have account log-in in future. But for now, having profile crawler fixed is more important. No rush =)

-Cheers.

Last edited by Mercyguy; 07.02.2020 at 19:36.
Reply With Quote
  #44  
Old 08.02.2020, 08:08
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

It does appear that profile and media scraping is working or on the right track of working.

I would like to report one issue before this goes on further.

The filenaming of downloaded files is a bit off for me. It's not a terrible issue, but it's a change from how Jdownloader previously handled naming of downloads.

So for example, if I scraped and downloaded this twitter post while doing a profile scrape

**External links are only visible to Support Staff****External links are only visible to Support Staff**

The file would normally in the past have come out as 1226027388355526658_EQO5aPkVUAIqQEH

The first set of numbers relating to the numbers in the URL and the last string of letters and digits relating to the file.

I would appreciate if I could have the option to set that naming scheme again as a great coincidence of that naming scheme is that post numbers on Twitter are always in ascending order. So it puts images and videos in chronological order in my folders.

Final note though, it doesn't appear as though this issue affects videos. Videos still get the URL_FileName naming scheme as before, so that may be worth looking into.
Reply With Quote
  #45  
Old 08.02.2020, 08:10
Telephoton Telephoton is offline
I will play nice!
 
Join Date: Jan 2020
Posts: 4
Default Twitter is downloading thumbnails instead of videos

Just as the title says. Jdownloader is downloading thumbnails/images/jpg instead of videos
Reply With Quote
  #46  
Old 09.02.2020, 02:39
Telephoton Telephoton is offline
I will play nice!
 
Join Date: Jan 2020
Posts: 4
Default

Jdownloader appears to be downloading jpg instead of videos for me. Maybe that is related to the way videos are named as mentioned in the above reply? It seems that images work ok though. I've tried renaming the jpg files to mp4 but that doesn't work because the files that are downloaded are actually jpg files. Apparently it's been doing that for the past few days and all the videos I downloaded are lost.

A post was made a few days or a week ago about videos not working, after which it seemed that the video downloading worked again but I guess it was just downloading jpg files the entire time.
Reply With Quote
  #47  
Old 10.02.2020, 09:06
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

In the interest of double checking your claim (cause I thought I was downloading videos just fine on my end) I have found that I am also suffering the same issue.

For testing purposes, I'm using this profile to test

**External links are only visible to Support Staff****External links are only visible to Support Staff**

10.02.20 02.55.45 <--> 10.02.20 03.06.28 jdlog://2925430900751/

And my logs
Reply With Quote
  #48  
Old 10.02.2020, 17:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,356
Default

Quote:
Originally Posted by Mercyguy View Post
Actually it's not true, Twitter have private accounts posts only visible to approved followers.
Account support did get Twitter accounts banned before which was of course not our intention.

If you have private twitter URLs and an example account, PN me.


Merged twitter threads!

@Telephoton & Akasen
Fixed!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #49  
Old 16.02.2020, 08:06
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default Twitter Download Naming Scheme Changed

So I'm just gonna make a formal post about this odd issue.

Simply put, for the longest time Jdowloader would download twitter videos and images by a scheme of [Tweet-ID]_[Image-ID] or whatever you want to call it.

Basically, a download would come as "608048483195617280_CHA431wWcAAhp7l". Now it's just "CHA431wWcAAhp7l"

What I want is for the previous notation back, or at least the option for it. It was very helpful in organizing images by date since all tweets posted are in ascending order.
Reply With Quote
  #50  
Old 17.02.2020, 21:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,356
Default

Merged Twitter threads.

Hi Akasen,
please add example URLs and describe exactly (e.g. with real examples) how you'd like the filenames to look like.

The filenames have not been changed intentionally - this was the side-effect of the recent twitter website changes.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #51  
Old 18.02.2020, 09:25
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Certainly.

So for example, if this post was downloaded (**External links are only visible to Support Staff****External links are only visible to Support Staff** )

what I would like is for the file to be named "1197122159883063296_EJ0IPocUUAEhz9M" and "1197122159883063296_EJ0IPofU8AANM3". This was how it was previously for some time now.

With regards to videos and anything downloaded as an MP4, the file would simply be just the URL twice over. So something like "921155373578506241_921155373578506241" **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #52  
Old 18.02.2020, 15:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,356
Default

I've updated the filenames accordingly for the next update

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #53  
Old 19.02.2020, 06:27
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Much appreciated!
Reply With Quote
  #54  
Old 22.02.2020, 08:43
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Hi! I have one other issue now.

Previously Jdownloader could take the specific link for a Twitter's Media tab and scrape ONLY the media. Now it doesn't seem to do just that anymore and scrapes the whole Twitter feed regardless. Any chance this could be looked into?

Twitter link for example
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #55  
Old 24.02.2020, 15:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,356
Default

What do you mean by "only" the media?
It grabs pictures and videos at the moment - how do you define "media"? What gets crawled that you don't wand?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #56  
Old 25.02.2020, 05:59
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Twitter has a tab for only media, that is the media a user has uploaded.

When you put in the URL the form of "**External links are only visible to Support Staff**, Jdownloader use to only scrape just THAT section of the twitter feed, thus limiting the scraping to the original content and uploads that user uploaded.

Last edited by Akasen; 25.02.2020 at 08:46.
Reply With Quote
  #57  
Old 25.02.2020, 14:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,356
Default

You are indeed right.

Try again after the next update.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #58  
Old 26.02.2020, 13:12
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

So just to double check something here and make sure I'm not complaining about an old issue that we've gone over is difficult to fix, and also that I'm not complaining about a problem when a plugin hasn't updated yet, I have a problem to throw out there again.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

26.02.20 06.58.01 <--> 26.02.20 07.04.10 jdlog://0080715302851/

Two sets of links and a log that should have my scraping and downloading of them. The first link there shows that the user uploaded media should be 860 photos/videos, but only picking up on 187. Second ones weird and Twitter says 93, but Jdownloader picked up 118, I just assume that Twitter counts single posts as a "photo/video post" and not the actual number of images contained. (EDIT: Okay, that's actually right. I'm over thinking that second ones numbers)

Anyways, my question is this: with the first one am I simply just experiencing that issue where Jdownloader can only hit the Twitter API so many times on one page? Or was that potentially fixed before and broken again?

Last edited by Akasen; 26.02.2020 at 13:15.
Reply With Quote
  #59  
Old 26.02.2020, 17:13
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,356
Default

@Akasen
Indeed media handling was still not working property!
I've fixed it with the next update and added additional loggers so I can easily see "how far" the crawl process is.
Your 2nd URL got me a total of 1117 items.


Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #60  
Old 27.02.2020, 04:17
Akasen Akasen is offline
Super Loader
 
Join Date: Jul 2017
Posts: 26
Default

Quote:
Originally Posted by pspzockerscene View Post
@Akasen
Indeed media handling was still not working property!
I've fixed it with the next update and added additional loggers so I can easily see "how far" the crawl process is.
Your 2nd URL got me a total of 1117 items.


Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!


-psp-
Woah, weird, that's a totally different result than on my end, huh.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 04:56.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.