JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 09.03.2020, 22:27
UriF UriF is offline
DSL User
 
Join Date: Oct 2013
Posts: 36
Default Site grabber 8muses.com

Ir it possibly to grab all of links from web site like this one?
**External links are only visible to Support Staff****External links are only visible to Support Staff**

For example I would like to get all of zip, rar, and, jpeg files which attached to this web site?

I tried JDownloader but link grabber could get some images and link but I am not able to get attachments lie IDM site grabber

Thanks


I installed JDownloader extension but it don't pick up attachment
**External links are only visible to Support Staff****External links are only visible to Support Staff**

return empty link

Last edited by UriF; 09.03.2020 at 22:47.
Reply With Quote
  #2  
Old 10.03.2020, 09:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Hi,

indeed we already have an 8muses crawler plugin.
I've added support for this kind of URLs (single pages only).

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 10.03.2020, 17:54
UriF UriF is offline
DSL User
 
Join Date: Oct 2013
Posts: 36
Default

Of course, I need 1 page only crawler. I don't want to buy/get IDM. I am totally satisfied with JDownoader.

Would I have to select "download all with JDownoader"? That's it?

Thanks
Reply With Quote
  #4  
Old 10.03.2020, 18:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Quote:
Originally Posted by UriF View Post
Would I have to select "download all with JDownoader"? That's it?
What do you mean?
OInce you get the update (read the linked FAQ!), you could easily add such URLs by just copyign the URL --> As long as the clipboard observation is activated.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #5  
Old 10.03.2020, 18:01
UriF UriF is offline
DSL User
 
Join Date: Oct 2013
Posts: 36
Default

Is it a problem to get a crawler to get images from other sites @ 8muses like imageshark like Chrome extension like one click downloader?

Than
Reply With Quote
  #6  
Old 10.03.2020, 18:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Well generally, every site is diffrent so it would either require another plugin, another link crawler rule (see forum search) or even browser addons / external scripts.

However, your request sounds as if you simply mean other parts of 8muses which could also easily be included in our already existing plugin (?)
Do you have any example URLs?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 10.03.2020, 19:33
UriF UriF is offline
DSL User
 
Join Date: Oct 2013
Posts: 36
Default

I mean like this
**External links are only visible to Support Staff****External links are only visible to Support Staff**

tabnails are linked to imagetwist.com

That's what this extension is doing. I suggest that every image host do it in different way????

https://chrome.google.com/webstore/d...dgakohog?hl=en
Reply With Quote
  #8  
Old 11.03.2020, 12:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

I've updated the crawler once again to grab this kind of content too.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 12.03.2020, 20:24
UriF UriF is offline
DSL User
 
Join Date: Oct 2013
Posts: 36
Default

I installed JD add-on (download/download with) about a week ago.

After today's JD update I am not able to download any file directly from 8muses site. I tried to download file or open fine using default archiver but I got error in Firefox download list (ctrl+J). After I removed this extension I am able to do it again.

By the way I tried to download files and copied/pasted from 8 muses site and I got list of jpg and rar files in JD link grabber. Thanks
Reply With Quote
  #10  
Old 16.03.2020, 02:39
netgearjd netgearjd is offline
Giga Loader
 
Join Date: Aug 2014
Posts: 92
Thumbs down

I don't have the JD add-on installed but this update seems to have broken forum downloads completely.

For example, take this thread: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Now I left-click, drag and select the 15 ZIP parts (.zip.001 through .zip.015) in Firefox and press Ctrl+C, then open JD's Add Links dialog and it correctly shows the 15 links as follows:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

However when I click Continue the Linkgrabber update icon just goes round and round and eventually stops without finding even a single file to download, despite the fact that these are direct HTTPS links! Heck, I can paste them all in a text file and grab them with wget or curl! Before the update I remember that JD used to display a "Do you want to do a deep scan" or similar message and then used to be able to find every single file, but now, nothing at all. Clearly the update messed something up big time.

Please fix this ASAP, thanks!
Reply With Quote
  #11  
Old 16.03.2020, 12:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Quote:
Originally Posted by netgearjd View Post
Please fix this ASAP, thanks!
We are a free software - please be patient.

Fixed!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #12  
Old 18.03.2020, 00:37
netgearjd netgearjd is offline
Giga Loader
 
Join Date: Aug 2014
Posts: 92
Exclamation

ASAP = As Soon As Possible, so of course it means I am patient... However it was disappointing to see the update breaking what was working perfectly before for years.

Thanks a lot for fixing it; I'll get back to you in case I face any more issues with JD and this particular site.
Reply With Quote
  #13  
Old 18.03.2020, 12:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Quote:
Originally Posted by netgearjd View Post
ASAP = As Soon As Possible, so of course it means I am patient... However it was disappointing to see the update breaking what was working perfectly before for years.
Happens.
It was my fault - I did not test it good enough

Quote:
Originally Posted by netgearjd View Post
Thanks a lot for fixing it; I'll get back to you in case I face any more issues with JD and this particular site.
No worries

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #14  
Old 21.03.2020, 23:52
netgearjd netgearjd is offline
Giga Loader
 
Join Date: Aug 2014
Posts: 92
Exclamation

Ok, so another regression... :(

Go to any post with embedded images (not attached files as in my report above), for example **External links are only visible to Support Staff****External links are only visible to Support Staff**, mark (select with the mouse) all the images, Ctrl+C and add links to JD. Linkgrabber again finds nothing at all to download.
Reply With Quote
  #15  
Old 23.03.2020, 13:07
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Quote:
Originally Posted by netgearjd View Post
Ok, so another regression... :(
Nope, basically the same cause as the other issue just that I did now know all kinds of direct-URLs they have so I was unable to exclude all from entering our crawler thus resulting in your mentioned issues.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #16  
Old 28.03.2020, 17:06
netgearjd netgearjd is offline
Giga Loader
 
Join Date: Aug 2014
Posts: 92
Default

More problems... :(

Check out the first post in **External links are only visible to Support Staff****External links are only visible to Support Staff**

If I select and copy all the 18 zip files (attachments), the Linkgrabber is able to find all 18 files after a deep link analysis, so that's still working fine after the fix.

However try selecting and copying the 6 preview images. The Add Links dialog displays the following:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

The problem is that the Linkgrabber displays only the 6 JPGs for downloading, but those are tiny low quality thumbnails generated by the forum! What we need instead are the 6 original high quality PNGs instead.

Please check the old code because JD never had any problems distinguishing between original embedded images and thumbnails earlier.
Reply With Quote
  #17  
Old 30.03.2020, 12:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Hi,

there is no problem or "new problem".
JDownloader does by default only pickup URLs with known filke-extension.
Not all of yours have a file-extension.
There are two ways to easily get your URLs added:
1. Simply add the main forum/thread URL to JD and our specified parser should find everything in a few seconds.
or
2. Add this LinkCrawler rule to JDownloader to force JD to recognize these forum URLs as direct-URLs:
Code:
[ {
  "enabled" : true,
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "name" : "8muses.com forum directurls example rule",
  "pattern" : "https?://(www\\.)?8muses\\.com/forum/attachments/.+",
  "rule" : "DIRECTHTTP",
  "packageNamePattern" : null,
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
} ]
See Settings --> Advanced Settings --> Link Crawler rules

EDIT
Not required anymore, see THIS post, point one.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 02.04.2020 at 12:04.
Reply With Quote
  #18  
Old 01.04.2020, 18:42
netgearjd netgearjd is offline
Giga Loader
 
Join Date: Aug 2014
Posts: 92
Exclamation

I honestly don't want to start an argument, but it is clearly a 'new' problem in the sense that (as I've mentioned more than once) none of these issues were present before the changes that were made in response to UriF's request. JD had no problems whatsoever recognizing and downloading embedded original images (and not their thumbnails) as well as attached files of any type from that forum!

Adding the thread URL is not viable because I don't want every single image in a thread including those I don't want or have already downloaded to show up twice, thus wasting my time. All I want is for JD to reliably locate and download the files (embedded or attached) that I have specifically selected and copy-pasted.

I tried the rule, and while it does make JD add the PNGs it of course means I have to get rid of all the JPGs. And you can very well imagine the problems when I have hundreds of JPGs selected in a thread, which will require separating out and deleting the thumbnail JPGs and leaving behind only the original JPGs.

I am making a sincere request, is there not any way to make the code accept only URLs containing "8muses.com/forum/attachments/" and reject those containing "8muses.com/forum/data/attachments/", like JD clearly did before?
Reply With Quote
  #19  
Old 02.04.2020, 11:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 52,519
Default

Hi again,

Please don't understand me wrong I'm not trying to hide my mistakes here - I'm still trying to help you
I've just locally reverted our plugin 'EightMusesComDecrypter' to the revision 40396 from 2019-02-21 to make sure I'm not talking trash here.

With the old revision, the only big difference is that there was no forum-thread crawler (has nothing todo with your direct-URLs).
--> This means I'm not talking trash - maybe their URLs have changed or they way you are adding them.
Our code is public which means you can easily check my statements.
* Yes, the issues with direct-URLs in between recent updates were caused by ma changes but your .jpg "issue" definitely not.

Your issues are caused by this situation:
- JD only accepts URLs with a file extension by default
- The ones you do NOT want contain a file extension
- The ones you want do not contain a file extension thus JD will not pick them up by default --> Link Crawler rule or plugin needed

To solve your issues, the following can be done:
1. Add plugin support for your URLs which do not contain an extension --> Then these would work out of the box for all users and you would not need a link crawler rule anymore.
I've done this with the next update!

2. Next you will still have the issue of your .jpg thumbnails which you do not want getting picked up.
This is not really a bug as JD will pickup any URL which it thinks is downloadable by default.
Sure I could display them as offline by default but then again maybe other users want to download them or maybe this type of URL will lead to different content in the future

You can however simply solve this by creating a LinkGrabber rule for such URLs to ignore them so you do not even get to see them.
Based on the information I have, all of these are the ones you never want to have:
Code:
8muses.com/forum/data/attachments/BLA
I could also filter out all .jpg files from 8muses but this may cause false positives in the future (well anything could cause false positives if they e.g. change their URL structure but usually this should last for a long time).
So here is the LinkGrabber Filter Rule you can use:

Once added and activated, add your "6 links" of your last post again.
Now you will only see 6 .png files in the LinkGrabber.
On the bottom corner there will be a button saying "Restore 6 filtered Links" --> These are the thumbnails you do not want.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 02.04.2020 at 12:04.
Reply With Quote
  #20  
Old 07.04.2020, 03:04
netgearjd netgearjd is offline
Giga Loader
 
Join Date: Aug 2014
Posts: 92
Thumbs up

Quote:
Originally Posted by pspzockerscene View Post
I'm still trying to help you
Thanks, I really do appreciate it!

Quote:
Originally Posted by pspzockerscene View Post
* Yes, the issues with direct-URLs in between recent updates were caused by ma changes but your .jpg "issue" definitely not.
Mate, I don't know what to say except to reiterate that unless I'm losing my mind, I don't recall having problems downloading embedded images from that forum (without the thumbnails butting in!) using JD. Perhaps as you say the link format changed in the forum around the same time you made your updates? Anyway, let's forget what happened and move on now...

Quote:
Originally Posted by pspzockerscene View Post
1. Add plugin support for your URLs which do not contain an extension --> Then these would work out of the box for all users and you would not need a link crawler rule anymore.
I've done this with the next update!
Awesome!

Quote:
Originally Posted by pspzockerscene View Post
I could also filter out all .jpg files from 8muses
That would be way overkill for sure!

Quote:
Originally Posted by pspzockerscene View Post
So here is the LinkGrabber Filter Rule you can use:
Once added and activated, add your "6 links" of your last post again.
Now you will only see 6 .png files in the LinkGrabber.
On the bottom corner there will be a button saying "Restore 6 filtered Links" --> These are the thumbnails you do not want.
Excellent, just tested and this is entirely satisfactory. Again, thanks and I hope that does it for this site for a while at least.

One last question - while getting rid of that Linkcrawler rule I saw a bunch of others with names like "Learned file extension", "Learned php script download" etc. Does JD add rules on its own based on patterns it notices?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 09:12.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.