JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 05.11.2023, 07:18
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default Very difficult (for me)filtering question 3d-model.org

instead of censoring, i removed the images, anyway instead of pasting the page url and have jd, grapb what ever links from that container, i specified the line numbers in the page source so that line 7-9 in page source is for item title, the first url that was regexed matched in line numbers 27-15 are preview url links, and so on. then i had the script replace and place holder texts to create final jsons, saved as crawljobs.

tough job for a newbie like me for sure.

Last edited by I3ordo; 21.11.2023 at 11:45. Reason: censoring was not effective
Reply With Quote
  #2  
Old 05.11.2023, 10:30
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

a fltering sort of "if filename is NOT similar to source url)...how similar right?...
Reply With Quote
  #3  
Old 07.11.2023, 12:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

I think I understood what you want but your attempt will not work since JDownloaders filtering capabilities do not allow for more complex if-else conditions like "if field X contains <thisVariable> and Y does not contain <thisVariable> then do Z".

I'd recommend going to other way:
Write a LinkCrawler rule for that website which will (hopefully) only return what you want ("whitelist attempt") then you will not need any additional filters.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #4  
Old 08.11.2023, 18:35
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

well actually i am instead going for crawljob route, i made a parser that scan the page source and get the urls i want and have the script prepare .crawljob files in json structure but , i just cant get the folderwatch add them properly. the fodlerwatch just moves the detected "crawljob file" to the "added" inside the "Folderwatch"
(despite the urls being allowed through link filters and appear in link grabber when manually paste the links)

here is my crawljob, cant see any json structure errors and jd is completely silent. while moving the job to the sub folder"added"

could not upload the .crawljob file as it is, forum uploader says "invalid file" so changed the extension to txt
Attached Files
File Type: txt Pottery Barn Shayne and Isabella white.txt (582 Bytes, 1 views)

Last edited by I3ordo; 08.11.2023 at 23:35.
Reply With Quote
  #5  
Old 09.11.2023, 10:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

Doesn't look like you really understood the basic concept of FolderWatch.
I'd still recommend creating one LinkCrawler rule instead but in the end it doesn't matter how you add the links to JD so if you got some external script which extracts the links and puts them in the crawljob this may seem unnecessarily complicated to me while it may still get the job done.

I'll still tell you what you need to do to fix that .crawljhob though:
1. Json structure is wrong!
You can validate json structures for example using the following webtool: jsoneditoronline.org
Putting multiple crawljobs into one file is possible but not like this -> See my example at the end of this post.

2. Unneeded fields.
While you can make use of the "filename" field, all you did is to put in the filename which JD would find later anyways -> You can lust remove that field.

Working example:
pastebin.com/raw/4kcK4CpP
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #6  
Old 13.11.2023, 01:10
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

1: You are right, structure was wrong and good to know that site if i ever have to use
2: At first thought, i definitely needed to define filename of image and zip but on a second thought since i was able to cherry pick the urls to be grabbed with the parsing script, i could make jd do it and it would have been much less painfull but i guess at the time i was keen on doing it through script just to see if i can... it was hard ! on a third thought! since i was feeding the url links not through a page url but separately as individual links, i definitely needed to define filenames for the crawljobs.
3:but the refence page for folderwatch had no example of a two item json structure at jsons....https://support.jdownloader.org/Know...basic-usage/23 so i here i was...

i one thing i am still wondering about this is how to set the folderwatch to a custom location....tried pasting a folder path with double quotes but it kept reverting to default... so i am using that still..

Last edited by I3ordo; 13.11.2023 at 01:13.
Reply With Quote
  #7  
Old 13.11.2023, 14:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

Quote:
Originally Posted by I3ordo View Post
but the refence page for folderwatch had no example of a two item json structure at
An example with two rules should not be needed because if you a bit about jsn structures, you can immediately see that this is an array structure so multiple elements are possible.
However I will update the example inside the article within the next few minutes to contain two crawljobs.

Quote:
Originally Posted by I3ordo View Post
i one thing i am still wondering about this is how to set the folderwatch to a custom location
Do you mean the folder which should be watched or the download folder included inside your folder watch rules?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #8  
Old 13.11.2023, 19:50
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

Quote:
Originally Posted by pspzockerscene View Post
Do you mean the folder which should be watched or the download folder included inside your folder watch rules?
the one that should be watched, if there can be two folders tha can be watched and how to setup either one or two but on a different drive. whenever i modify , it reverts it to "folderwatch" located in "\AppData\Local\JDownloader 2.0\folderwatch"
Reply With Quote
  #9  
Old 14.11.2023, 11:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

Where did you try to modify the setting?
In the GUI or in the advanced settings?
Which values did you try to add? Please post them [censor private data inside your file-paths].

Keep in mind that the setting is a json structure so you need to use proper escaping.
Consider to use json editors like for example jsoneditoronline.org to edit json structures.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #10  
Old 21.11.2023, 11:40
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

in advanced settings menu (GUI)
tried this
[{"folderwatch":"d:\\some folder"}] (no errors mentioned in json site)
but does not accept.

also tried this;
[
{
"folderwatch": "d:\\some folder"
}
]
nope...
[{"folderwatch":"d:\\some folder"}]
{"folderwatch": "d:\\some folder"}
The documentation is awesome for newbies...(just one tiny example would be great to get going) but i love JD!

Last edited by I3ordo; 21.11.2023 at 12:01.
Reply With Quote
  #11  
Old 21.11.2023, 13:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

Quote:
Originally Posted by I3ordo View Post
but does not accept.
Most likely you just clicked somewhere else.
Edit the field and then hit enter. Works fine here.
EDIT
I've created a ticket for this as I think that the current behavior is neither good nor intended:


Quote:
Originally Posted by I3ordo View Post
The documentation is awesome for newbies...(just one tiny example would be great to get going) but i love JD!
I guess you mean "awful".
I parially agree but we do not have time for this. You are free to add documentation / help articles yourself.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 21.11.2023 at 13:17.
Reply With Quote
  #12  
Old 21.11.2023, 14:21
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,515
Default

Quote:
Originally Posted by I3ordo View Post
tried this
[{"folderwatch":"d:\\some folder"}] (no errors mentioned in json site)
but does not accept.
Do not use the key name ("folderwatch"). Type only the path to folder. Comma separated for multiple folders.

Code:
[ "c:\\folder one", "d:\\folder two"]
Reply With Quote
  #13  
Old 22.11.2023, 04:16
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

Quote:
Originally Posted by mgpai View Post
Do not use the key name ("folderwatch"). Type only the path to folder. Comma separated for multiple folders.

Code:
[ "c:\\folder one", "d:\\folder two"]
oh this makes sense to me too now! thanks
Reply With Quote
  #14  
Old 22.11.2023, 04:22
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

i think i will suggest something actually good, the crawljobs are missing a key feature. the container info. i have checked twice but could not find info about setting a container, the web page that those links gathered from.

99% of the usage scenarios, there is no need to have a container info in a crawljob i assume, but as 1% here, hi! think having the container fieldname would allow me to visit the page that those links are residing in. currently the implementation does not look great either. if i check the download urls of the files that were fed by the crawljob files, the container points to the local crawljob file which makes sense but the actual web page info is more beneficial for this case. you can go directly to web page and see whats wrong without having to start an investigation on your local pc...

i dont think, using the "comment" field as a place for placing the container url would work

Last edited by I3ordo; 22.11.2023 at 10:17.
Reply With Quote
  #15  
Old 22.11.2023, 12:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

Quote:
Originally Posted by I3ordo View Post
i think i will suggest something actually good, the crawljobs are missing a key feature. the container info. i have checked twice but could not find info about setting a container, the web page that those links gathered from.
Please provide a full real life example where you think adding an additional "container" URL / source-URL would make sense.

Quote:
Originally Posted by I3ordo View Post
currently the implementation does not look great either. if i check the download urls of the files that were fed by the crawljob files, the container points to the local crawljob file which makes sense but the actual web page info is more beneficial for this case.
That is just the default setting. You can get the link to the file, see this support article:
https://support.jdownloader.org/Know...iew-added-urls

Quote:
Originally Posted by I3ordo View Post
i dont think, using the "comment" field as a place for placing the container url would work
If it's just for you as a reminder as to where you got the URLs from I do not see any issue when using the comment field as it's usually empty.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #16  
Old 22.11.2023, 22:35
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

container url was successfully added as a "comment", with this work around, i can visit the container url by copying it from the comment field.
HTML Code:
[
  {
    "comment": "**External links are only visible to Support Staff**,
    "text": "**External links are only visible to Support Staff**,
    "filename": "TV Wall Modern.webp",
    "packageName": "somesite-crawl",
    "downloadFolder": "W:\\!_some_Workshop\\__somesite-org"
  },
  {
    "comment": "**External links are only visible to Support Staff**,
    "text": "**External links are only visible to Support Staff**,
    "filename": "TV Wall Modern.zip",
    "packageName": "somesite-crawl",
    "downloadFolder": "W:\\!_some_Workshop\\__somesite-org"
  }
]
"That is just the default setting. You can get the link to the file, see this support article:"
yes i knew that but since crawljob don't have a "container"field(it s called container, when i paste a web page that contains the file urls, when i use a crawljob, container url is the file hoster's link so a bit of confusion there too) and uses the filepath of the crawljob as container url(source url), it does not make sense to me , having the actual container url from a json would allow me to ctr+w to the web page.

btw here is the part of the script that replaces and creates the crawljobs.
HTML Code:
# Updated JSON structure template
json_structure_template = [
    {
        "comment": "model_url",
        "text": "preview_url",
        "filename": "{preview_filename}",
        "packageName": "somesite-crawl",
        "downloadFolder": "W:\\!_some_Workshop\\__model-site"
    },
    {
        "comment": "model_url",
        "text": "archive_url",
        "filename": "{archive_filename}",
        "packageName": "somesite-crawl",
        "downloadFolder": "W:\\!_some_Workshop\\__model-site"
    }
]
And the part that replaces the placeholders,
Code:
# Function to replace placeholders in the JSON structure with actual data
def replace_placeholders(model_title, archive_url, preview_url, model_url):
    # Replace special characters in the model title
    model_title = model_title.replace("&", " and ")
    model_title = model_title.replace("№", "No.")

    # Create a copy of the original JSON structure for each URL
    json_structure = copy.deepcopy(json_structure_template)

    for entry in json_structure:
        entry["comment"] = model_url  # Add the actual model URL to the "comment" field
        if entry["text"] == "preview_url":
            entry["text"] = preview_url
            entry["filename"] = f"{model_title}.webp"
        elif entry["text"] == "archive_url":
            entry["text"] = archive_url
            entry["filename"] = f"{model_title}.zip"

    return json_structure
I somewhat achieved what i needed but my suggestion still stands as "it would be nice"
Attached Thumbnails
Untitled-1.jpg  

Last edited by I3ordo; 22.11.2023 at 22:46.
Reply With Quote
  #17  
Old 23.11.2023, 11:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

None of your examples are real life examples with working links.
I think I still get what you want.

This is my assessment:
Having a crawljob as source field is not any important source which gets used by plugins later on so I think it is possible to change the handling to allow the user to override the source field for crawljobs with a custom value.
I've created a ticket though I won't provide any ETA and if development pace stays like it is now, this will never be worked on.
Ticket:


All in all your suggestion is a niche suggestion and I think using the comment field for said source information should be fine.
You can even add custom hotkeys to copy the content of the source field so it should be super easy to get to this information again.

Possible workarounds:
You might be able to change the source URL by using an EventScripter script. I recommend looking into this option:
EventScripter forum thread:
https://board.jdownloader.org/showthread.php?t=70525
EventScripter help article:
https://support.jdownloader.org/Know...event-scripter
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #18  
Old 23.11.2023, 15:48
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,515
Default

@I3ordo,

You can also check 'flashgot' API. It has option to specify source urls (origin, referrer etc.), which can send links to JD using HTTP requests, without the need of creating/adding crawljobs.
Reply With Quote
  #19  
Old 23.11.2023, 16:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,789
Default

Flashgot docs:
https://support.jdownloader.org/Know...hgot-interface
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #20  
Old 29.11.2023, 15:52
I3ordo I3ordo is offline
Mega Loader
 
Join Date: Mar 2022
Posts: 65
Default

i got what the task done by creating a crawler job creator script and maybe it is an overkill but it is working great atm, (knock on wood) wonder if flashgot would be able to get only some of the links from the page source by restricting to get urls from specific lines..
like "look at page source, line 6 find something beginning with "somehoster url"
maybe it did and i unnecessarily had to learn a lot of stuff not knowing that...

anyway thank you for the interest in this topic.

I can only suggest a small improvement for JD in the comment field of the downloads...

See i am using comment field for storing the container urls, and JD could detect and change the behaviour of the comment , and make it clickable link if it s an url...
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:28.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.