JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #61  
Old 18.11.2022, 18:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Quote:
Originally Posted by mwl View Post
I might have some questions as I go, so what would be the best way to reach out to devs for some answers while I'm working?
best by mail support@jdownloader.org but be aware at the weekend I rarly check those

Quote:
Originally Posted by mwl View Post
I'll take a look at some other API clients to see what they can and can't do.
as far as I remember, the api was limited in features/posibilites in the past but I don't know about current api. sure, feel free to test other tools and check/compare their feature-set against JDownloader.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #62  
Old 18.11.2022, 18:45
mwl mwl is offline
JD Fan
 
Join Date: Mar 2013
Posts: 77
Default

Quote:
Originally Posted by Jiaz View Post
"Should" not be necessary to regulary reset it as you don't have to regulary re-login at website.
Like I said, my favourite word.

They're almost certainly locking it out at the server side, since while it's running I occasionally get 403s in the browser.

Anyway, it's good to at least have something working again.
Reply With Quote
  #63  
Old 18.11.2022, 18:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Quote:
Originally Posted by mwl View Post
They're almost certainly locking it out at the server side, since while it's running I occasionally get 403s in the browser.
To be fair, this highly depends on workload and user case. Doing few requests/downloads is complete different than (some ppl prefer to) download/mass crawl/create complete clone of sites/profiles
__________________
JD-Dev & Server-Admin
Reply With Quote
  #64  
Old 18.11.2022, 18:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Quote:
Originally Posted by mwl View Post
Anyway, it's good to at least have something working again.
Absolute right. Better a working (but maybe limited) plugin that no support at all/complete broken plugin. Also, depending on use case, it might be better to use specialized tools/scripts for the job.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #65  
Old 21.11.2022, 17:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,044
Default

Quote:
Originally Posted by mwl View Post
In some cases, instead of the actual item, it's downloading a DA logo. If you're looking for some way to detect it, it's 10648 bytes.
This will be fixed with the next CORE-update.
Thanks for reporting.
Quote:
Originally Posted by Jiaz View Post
@mwl: thanks for the feedback. Maybe literature support broken/not yet supported. Maybe can provide example links?
As part of the fix mentioned above, this is also fixed.

Quote:
Originally Posted by mwl View Post
Also, there seems to be no way to craft a packagizer rule to differentiate between user galleries and favorites, as according to the rule tester both the Download URL and Source URL are the individual deviation URL rather than the gallery URL. I acknowledge that this might already be known.
What are you trying to achieve here?

Quote:
Originally Posted by mwl View Post
Out of interest, what was the reason for sticking with scraping rather than trying to use the API?
The Web-API is used by the plugin now.
Which API are you referring to?

Quote:
Originally Posted by mwl View Post
Anyway, it's good to at least have something working again.
Please keep in mind that it is still heavily work in progress.
I'm working on it every now and then when I'm finding time and I'm also relying on our user to provide feedback.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #66  
Old 23.11.2022, 16:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,044
Default

The mentioned CORE-updates have been released and we've optimized quite some stuff.

We're now waiting for your feedback regarding our partly rewritten Deviantart plugins.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #67  
Old 26.11.2022, 08:47
mwl mwl is offline
JD Fan
 
Join Date: Mar 2013
Posts: 77
Default

Copying a gallery link to the clipboard does not crawl the gallery. Adding it manually in LinkGrabber reuslts in crawling the entire page rather than just the gallery entries.

26.11.22 06.41.34 <--> 26.11.22 06.43.39 jdlog://9141311370661/
Reply With Quote
  #68  
Old 26.11.2022, 16:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

@mwl: Whenever you copy a link to clipboard and nothing happens, it is a strong sign that the site/link format is not yet supported by JDownloader. When you manually add/paste it and then links show up, than it's a 100% sign that the site/link format is not supported and deep decrypt tries to find anything that can be downloaded.
Please provide example links that you try to add, then we can check/update plugin to support those as well
__________________
JD-Dev & Server-Admin
Reply With Quote
  #69  
Old 30.11.2022, 00:49
mwl mwl is offline
JD Fan
 
Join Date: Mar 2013
Posts: 77
Default

This is a session that should have nothing else (apart from one stray download).

Log (should be in debug mode):
29.11.22 22.43.36 <--> 29.11.22 22.48.04 jdlog://6051311370661/

Attemped links:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Hope these are useful.
Reply With Quote
  #70  
Old 01.12.2022, 19:19
ecclessia ecclessia is offline
Junior Loader
 
Join Date: Jun 2019
Posts: 14
Default

Good day.

It looks like plugin generally works, yet there are some pictures, which don't have it's download button, yet there is information about them actually being way higher resolution, which current version of plugin cannot process.

I had found some console app, that somehow can access API and does download full resolution versions - **External links are only visible to Support Staff****External links are only visible to Support Staff**

Also, when i try to use cookies to log in, i get logged out relatively quickly (about 10-15 or so minutes), so i have to relog and readd cookies to account.
Reply With Quote
  #71  
Old 01.12.2022, 19:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

@ecclessia: can you please provide example links? all my tested links, JDownloader was always able to fetch highest resolution/quality version.
Quote:
Originally Posted by ecclessia View Post
Also, when i try to use cookies to log in, i get logged out relatively quickly (about 10-15 or so minutes), so i have to relog and readd cookies to account.
Can you try to create a debug log for when this happens? see https://support.jdownloader.org/Know...d-session-logs
Enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader, now wait for issue to happen and then create log and post logID here
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 01.12.2022 at 20:04.
Reply With Quote
  #72  
Old 01.12.2022, 19:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

@mwl: just use /gallery link and remove the trailing /all. Those urls are not yet supported but will be with next update.
with next update also /gallery/scraps will be supported
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 01.12.2022 at 20:12.
Reply With Quote
  #73  
Old 01.12.2022, 22:19
ecclessia ecclessia is offline
Junior Loader
 
Join Date: Jun 2019
Posts: 14
Default

Quote:
Originally Posted by Jiaz View Post
and post logID here
01.12.22 22.58.13 <--> 01.12.22 23.16.19 jdlog://2451311370661/

Here you go. I had specifically tried to add long galleries, yet it was on the first one

**External links are only visible to Support Staff****External links are only visible to Support Staff**

within 15 minutes or so.

I hadn't seen logs myself, yet it might be due to me using vpn (no such problems with site without JDownloader though) or with great amount of request, even though i had tried to limit download speed to 100 kb/s.

And, yes, after JDownloader starts saying, that cookies had expires, i'm also automatically logged out of site as well.
Reply With Quote
  #74  
Old 02.12.2022, 15:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Quote:
Originally Posted by Jiaz View Post
@ecclessia: can you please provide example links? all my tested links, JDownloader was always able to fetch highest resolution/quality version.
Could you also please provide example links where you think a higher quality is available but not found by JDownloader?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #75  
Old 02.12.2022, 15:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Quote:
Originally Posted by ecclessia View Post
And, yes, after JDownloader starts saying, that cookies had expires, i'm also automatically logged out of site as well.
Same cookie as in JDownloader? or did you use different login/cookie session for JDownloader as in Browser?

@ecclessia:
Quote:
HTTP/1.1 403 Forbidden
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
So either this happens because of too many requests (crawl protection or overall request/interval limit) or cloudfront configuration issue.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 02.12.2022 at 16:36.
Reply With Quote
  #76  
Old 02.12.2022, 16:29
ecclessia ecclessia is offline
Junior Loader
 
Join Date: Jun 2019
Posts: 14
Default

Quote:
Originally Posted by Jiaz View Post
Same cookie as in JDownloader?
Well, as per provided within application instruction:
1. I had installed "Flag Cookies" Firefox addon, l
2. logged into deviantart site,
3. specifically reloaded page just in case,
4. opened addon preferences and copied cookies through use of button @Export cookie data to clipboard
5. inside JDownloader application i had opened account manager, opened account login/password edit window (login was already applied earlier)
6. inserted clipboard data into password field and applied it.

After check Jdownloader had indicated, that everything is ok (everything green, "free account" text), then i had added gallery to link parser, after gallery had been parsed i had started download. After a little more than 15 minutes file downloads had stopped starting with text "Account is missing", inside account manager was text "Cookies had expired" (or similar phrase).

When i had created this thread (when deviantart had just changed it's layout) there was no such problem.
Reply With Quote
  #77  
Old 02.12.2022, 16:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

@ecclessia: I don't think that the cookies are expired but rather temp/IP blocked
JDownloader tries to connect website but fails due to shown message above. JDownloader has no special *blocked* handling and thus just tells that login via cookies has failed.
Have you tried with fresh IP eg Reconnect/VPN? or wait some time. to me it looks more like IP blocked
as the block comes from cloudfront
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 02.12.2022 at 16:38.
Reply With Quote
  #78  
Old 02.12.2022, 16:57
ecclessia ecclessia is offline
Junior Loader
 
Join Date: Jun 2019
Posts: 14
Default

Quote:
Originally Posted by Jiaz View Post
Have you tried with fresh IP eg Reconnect/VPN? or wait some time. to me it looks more like IP blocked
It might be, though i think it is more of "let's disable this cookie session for this spammer", so cookie becomes invalid, as relogging into site and getting new cookies into jd allows to continue for same time period.

Is there any way to put some timeout between downloads? I had tried to limit downloads to 1 at a time and to limit download speed, but that didn't help.
Reply With Quote
  #79  
Old 02.12.2022, 17:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Quote:
Originally Posted by ecclessia View Post
It might be, though i think it is more of "let's disable this cookie session for this spammer", so cookie becomes invalid, as relogging into site and getting new cookies into jd allows to continue for same time period.
You still can open the website? JDownloader only receives the 403 blocked cloudfront website.
Quote:
Originally Posted by ecclessia View Post
Is there any way to put some timeout between downloads? I had tried to limit downloads to 1 at a time and to limit download speed, but that didn't help.
You could use Eventscripter script for that, search https://board.jdownloader.org/showthread.php?t=70525
as far as I know there is a script that waits x seconds after each download. For example see https://board.jdownloader.org/showpo...postcount=2319
Once we know what *limits* there are or what might help to avoid this, then we can update plugin accordingly
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 02.12.2022 at 17:21.
Reply With Quote
  #80  
Old 02.12.2022, 17:21
ecclessia ecclessia is offline
Junior Loader
 
Join Date: Jun 2019
Posts: 14
Default

Quote:
Originally Posted by Jiaz View Post
You still can open the website? JDownloader only receives the 403 blocked cloudfront website.
Sometimes during downloads still being active site does throw 403 error (and stil from time to time opens with account still being logged in), but right after downloads had stopped (or, maybe, several seconds later) site definitely opens normally, but logged out of account.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 07:22.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.