#1
|
|||
|
|||
Site grabber 8muses.com
Ir it possibly to grab all of links from web site like this one?
**External links are only visible to Support Staff****External links are only visible to Support Staff** For example I would like to get all of zip, rar, and, jpeg files which attached to this web site? I tried JDownloader but link grabber could get some images and link but I am not able to get attachments lie IDM site grabber Thanks I installed JDownloader extension but it don't pick up attachment **External links are only visible to Support Staff****External links are only visible to Support Staff** return empty link Last edited by UriF; 09.03.2020 at 22:47. |
#2
|
||||
|
||||
Hi,
indeed we already have an 8muses crawler plugin. I've added support for this kind of URLs (single pages only). Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3
|
|||
|
|||
Of course, I need 1 page only crawler. I don't want to buy/get IDM. I am totally satisfied with JDownoader.
Would I have to select "download all with JDownoader"? That's it? Thanks |
#4
|
||||
|
||||
What do you mean?
OInce you get the update (read the linked FAQ!), you could easily add such URLs by just copyign the URL --> As long as the clipboard observation is activated. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#5
|
|||
|
|||
Is it a problem to get a crawler to get images from other sites @ 8muses like imageshark like Chrome extension like one click downloader?
Than |
#6
|
||||
|
||||
Well generally, every site is diffrent so it would either require another plugin, another link crawler rule (see forum search) or even browser addons / external scripts.
However, your request sounds as if you simply mean other parts of 8muses which could also easily be included in our already existing plugin (?) Do you have any example URLs? -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
|||
|
|||
I mean like this
**External links are only visible to Support Staff****External links are only visible to Support Staff** tabnails are linked to imagetwist.com That's what this extension is doing. I suggest that every image host do it in different way???? https://chrome.google.com/webstore/d...dgakohog?hl=en |
#8
|
||||
|
||||
I've updated the crawler once again to grab this kind of content too.
Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#9
|
|||
|
|||
I installed JD add-on (download/download with) about a week ago.
After today's JD update I am not able to download any file directly from 8muses site. I tried to download file or open fine using default archiver but I got error in Firefox download list (ctrl+J). After I removed this extension I am able to do it again. By the way I tried to download files and copied/pasted from 8 muses site and I got list of jpg and rar files in JD link grabber. Thanks |
#10
|
|||
|
|||
I don't have the JD add-on installed but this update seems to have broken forum downloads completely.
For example, take this thread: **External links are only visible to Support Staff****External links are only visible to Support Staff** Now I left-click, drag and select the 15 ZIP parts (.zip.001 through .zip.015) in Firefox and press Ctrl+C, then open JD's Add Links dialog and it correctly shows the 15 links as follows: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** However when I click Continue the Linkgrabber update icon just goes round and round and eventually stops without finding even a single file to download, despite the fact that these are direct HTTPS links! Heck, I can paste them all in a text file and grab them with wget or curl! Before the update I remember that JD used to display a "Do you want to do a deep scan" or similar message and then used to be able to find every single file, but now, nothing at all. Clearly the update messed something up big time. Please fix this ASAP, thanks! |
#11
|
||||
|
||||
We are a free software - please be patient.
Fixed! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#12
|
|||
|
|||
ASAP = As Soon As Possible, so of course it means I am patient... However it was disappointing to see the update breaking what was working perfectly before for years.
Thanks a lot for fixing it; I'll get back to you in case I face any more issues with JD and this particular site. |
#13
|
||||
|
||||
Quote:
It was my fault - I did not test it good enough Quote:
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#14
|
|||
|
|||
Ok, so another regression... :(
Go to any post with embedded images (not attached files as in my report above), for example **External links are only visible to Support Staff****External links are only visible to Support Staff**, mark (select with the mouse) all the images, Ctrl+C and add links to JD. Linkgrabber again finds nothing at all to download. |
#15
|
||||
|
||||
Nope, basically the same cause as the other issue just that I did now know all kinds of direct-URLs they have so I was unable to exclude all from entering our crawler thus resulting in your mentioned issues.
Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#16
|
|||
|
|||
More problems... :(
Check out the first post in **External links are only visible to Support Staff****External links are only visible to Support Staff** If I select and copy all the 18 zip files (attachments), the Linkgrabber is able to find all 18 files after a deep link analysis, so that's still working fine after the fix. However try selecting and copying the 6 preview images. The Add Links dialog displays the following: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** The problem is that the Linkgrabber displays only the 6 JPGs for downloading, but those are tiny low quality thumbnails generated by the forum! What we need instead are the 6 original high quality PNGs instead. Please check the old code because JD never had any problems distinguishing between original embedded images and thumbnails earlier. |
#17
|
||||
|
||||
Hi,
there is no problem or "new problem". JDownloader does by default only pickup URLs with known filke-extension. Not all of yours have a file-extension. There are two ways to easily get your URLs added: 1. Simply add the main forum/thread URL to JD and our specified parser should find everything in a few seconds. or 2. Code:
[ { "enabled" : true, "updateCookies" : true, "logging" : false, "maxDecryptDepth" : 1, "name" : "8muses.com forum directurls example rule", "pattern" : "https?://(www\\.)?8muses\\.com/forum/attachments/.+", "rule" : "DIRECTHTTP", "packageNamePattern" : null, "passwordPattern" : null, "formPattern" : null, "deepPattern" : null, "rewriteReplaceWith" : null } ] EDIT Not required anymore, see THIS post, point one. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 02.04.2020 at 12:04. |
#18
|
|||
|
|||
I honestly don't want to start an argument, but it is clearly a 'new' problem in the sense that (as I've mentioned more than once) none of these issues were present before the changes that were made in response to UriF's request. JD had no problems whatsoever recognizing and downloading embedded original images (and not their thumbnails) as well as attached files of any type from that forum!
Adding the thread URL is not viable because I don't want every single image in a thread including those I don't want or have already downloaded to show up twice, thus wasting my time. All I want is for JD to reliably locate and download the files (embedded or attached) that I have specifically selected and copy-pasted. I tried the rule, and while it does make JD add the PNGs it of course means I have to get rid of all the JPGs. And you can very well imagine the problems when I have hundreds of JPGs selected in a thread, which will require separating out and deleting the thumbnail JPGs and leaving behind only the original JPGs. I am making a sincere request, is there not any way to make the code accept only URLs containing "8muses.com/forum/attachments/" and reject those containing "8muses.com/forum/data/attachments/", like JD clearly did before? |
#19
|
||||
|
||||
Hi again,
Please don't understand me wrong I'm not trying to hide my mistakes here - I'm still trying to help you I've just locally reverted our plugin 'EightMusesComDecrypter' to the revision 40396 from 2019-02-21 to make sure I'm not talking trash here. With the old revision, the only big difference is that there was no forum-thread crawler (has nothing todo with your direct-URLs). --> This means I'm not talking trash - maybe their URLs have changed or they way you are adding them. Our code is public which means you can easily check my statements. * Yes, the issues with direct-URLs in between recent updates were caused by ma changes but your .jpg "issue" definitely not. Your issues are caused by this situation: - JD only accepts URLs with a file extension by default - The ones you do NOT want contain a file extension - The ones you want do not contain a file extension thus JD will not pick them up by default --> Link Crawler rule or plugin needed To solve your issues, the following can be done: 1. Add plugin support for your URLs which do not contain an extension --> Then these would work out of the box for all users and you would not need a link crawler rule anymore. I've done this with the next update! 2. Next you will still have the issue of your .jpg thumbnails which you do not want getting picked up. This is not really a bug as JD will pickup any URL which it thinks is downloadable by default. Sure I could display them as offline by default but then again maybe other users want to download them or maybe this type of URL will lead to different content in the future You can however simply solve this by creating a LinkGrabber rule for such URLs to ignore them so you do not even get to see them. Based on the information I have, all of these are the ones you never want to have: Code:
8muses.com/forum/data/attachments/BLA So here is the LinkGrabber Filter Rule you can use: Once added and activated, add your "6 links" of your last post again. Now you will only see 6 .png files in the LinkGrabber. On the bottom corner there will be a button saying "Restore 6 filtered Links" --> These are the thumbnails you do not want. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 02.04.2020 at 12:04. |
#20
|
|||
|
|||
Thanks, I really do appreciate it!
Quote:
Quote:
That would be way overkill for sure! Quote:
One last question - while getting rid of that Linkcrawler rule I saw a bunch of others with names like "Learned file extension", "Learned php script download" etc. Does JD add rules on its own based on patterns it notices? |
#21
|
||||
|
||||
Thanks for your feedback.
Updates have been released multiple days ago which means everything should work as expected now Quote:
With your URLs which it did not pickup, this will not work because your URLs do not contain a file-extension at all so there is nothing for it to "learn" -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#22
|
|||
|
|||
Hey pspzockerscene, sorry to say but I am back...
The following post has 3 attachments - **External links are only visible to Support Staff****External links are only visible to Support Staff** Add Links dialog shows the following correctly: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** Any reason why LinkGrabber consistently fails to identify only the second link properly (part 2 of the RAR)? It shows up as "unknownFileName" and with an "Unknown" file size and naturally fails to download, but the exact same link works fine in a browser. |
#23
|
||||
|
||||
Hi,
your 2nd link states "You must be logged-in to do that" via browser. -psp- EDIT I guess the user who uploaded the content might have set this flag just by mistake - maybe ask him to remove that restriction
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 14.04.2020 at 12:27. |
#24
|
|||
|
|||
Stopped working again
Hello, just downloaded jdownloader last week and I love it so far. However I was using it on 8muses 1 night and literally the next night it stopped working for the site. Now it only finds the thumbnails and they have these impossibly long names for them.
|
#25
|
|||
|
|||
Forgot to add
I forgot to add that I tried adding that linkgrabber rule and I get the same results either with the rule or without the rule.
|
#26
|
||||
|
||||
Please provide example links and logs, see https://support.jdownloader.org/Know...d-session-logs
__________________
JD-Dev & Server-Admin |
#27
|
||||
|
||||
I'm unable to reproduce your issue with any of the testlinks in this thread.
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#28
|
|||
|
|||
Ok heres an example
**External links are only visible to Support Staff****External links are only visible to Support Staff** and is this how I show the log? 14.05.20 11.39.06 <--> 14.05.20 11.54.43 jdlog://0686915302851/ |
#29
|
|||
|
|||
**External links are only visible to Support Staff****External links are only visible to Support Staff** is being redirected to **External links are only visible to Support Staff****External links are only visible to Support Staff**
Please add the new domain changes. |
#30
|
|||
|
|||
So I added the new domain to the linkgrabber rule and I still get the same problem.
15.05.20 01.34.42 <--> 15.05.20 01.39.07 jdlog://3407915302851/ |
#31
|
||||
|
||||
Wait for next plugin update
__________________
JD-Dev & Server-Admin |
#32
|
|||
|
|||
Working after the latest update, thanks.
|
#33
|
|||
|
|||
working great thank you
|
#34
|
||||
|
||||
Thanks for the feedback!
__________________
JD-Dev & Server-Admin |
#35
|
|||
|
|||
8muses plugin?
Sorry to come into this so late, but I don't see a "plugin" for 8muses forum. Is there a different site I should pull this down from?
Thanks! |
#36
|
||||
|
||||
@Roger-that: Only plugins with settings are listed in Settings->Plugins.
What exactly is the problem?
__________________
JD-Dev & Server-Admin |
#37
|
|||
|
|||
Quote:
I can manage to download the attachments with "downloadthemall" but I suppose that's because it uses my logged on session from the browser. Is there a way to do this from Jdownloader2? |
#38
|
||||
|
||||
@Roger-that: We need username/password to add account support for this plugin. Please send details to support@jdownloader.org
__________________
JD-Dev & Server-Admin |
#39
|
||||
|
||||
Hi,
the update includes the following changes: - Added account support - Fixed online status for files for which an account is required to download - Fixed "temporary filenames" shown if no "real" filenames are available yet --> This especially works around the "unknownFileName" which would happen for files, only downloadable via account Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#40
|
|||
|
|||
Perfect! Thanks!
|
|
|