JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 11.05.2021, 06:22
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,528
Default

Quote:
Originally Posted by Wheezer View Post
Any suggestions ???
Could be Java version related. You'll have to wait for Jiaz. In the meanwhile, you can add it directly in Settings > Advanced Settings > EventScripter.scripts.

Code:
[{
    "eventTrigger": "LINKGRABBER_TABLE_CONTEXT_MENU_BUTTON",
    "enabled": true,
    "name": "Wheezer",
    "script": "/*\n    Set package name from container url\n    Trigger : Linkgrabber Context Menu Button Pressed\n    Customize context menu > Add a new \"Eventscriopter Trigger\" button > Rename it to \"Package from container url\"\n*/\n\nif (name == \"Package from container url\") {\n    var urls = {};\n\n    lgSelection.links.forEach(function(link) {\n        var url = link.containerURL;\n\n        if (url) {\n            urls[url] = urls[url] || [];\n            urls[url].push(link.UUID);\n        }\n    })\n\n    for (url in urls) {\n        callAPI(\"linkgrabberv2\", \"movetoNewPackage\", urls[url], [], url, null);\n    }\n}",
    "eventTriggerSettings": {
        "isSynchronous": false
    },
    "id": 1620634816265
}]
Reply With Quote
  #22  
Old 11.05.2021, 08:37
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

Ahhh .... always a backdoor.

Let me backup again, add the script and see what happens.

Cheers!
... W ...
Reply With Quote
  #23  
Old 11.05.2021, 09:51
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

Well I backed up JD added the script, restarted and ... << ??? >>

The script shows in the EventScripter:Scripts section, but not sure where it will show up so I can test it.

Sorry for being a bit dense on this.

Cheers!
... W ...
Reply With Quote
  #24  
Old 11.05.2021, 10:47
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,528
Default

Quote:
Originally Posted by Wheezer View Post
The script shows in the EventScripter:Scripts section, but not sure where it will show up so I can test it.
Once the script has been installed it will be listed in Settings > Eventscriipter. In the linkgrabber list, customize the context menu and create a button as specified in the script. A corresponding menu item will then be available in the context menu which can be used to select links/packages and execute the script.
Reply With Quote
  #25  
Old 11.05.2021, 11:24
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

BINGO ... got it working .. thanks for the help.

I experimented a bit with the context menu a bit so I figured out what you finally described. An odd thing is that now there is a script in the system, the editor comes up.

The first test on 1373 links worked, they were broken out into appropriate packages.
The second test also worked, 3688 links broken into 15 packages.
The link count is correct (same total in split packages as in the source package).

The actual processing is interesting.

When run there is a bit of a CPU spike as it preprocesses (15-18% CPU load) and nothing visible.
Then it starts to split the source package and CPU goes up to about 25%-27%.
When done, back to lesser load again. The process time and CPU load so far seems dependent on the number of links in the source package. Interesting to watch it work.

The only quirk it that one link group may not be renamed. NO big deal, looking at the "Downloaded from" column indicates that all the links are the same source URL, so just a simple single rename.

At this point I would say it is an unqualified success !!

Many thanks for the script and stepping me through this !

Cheers.
.. W ...

Last edited by Wheezer; 11.05.2021 at 11:37.
Reply With Quote
  #26  
Old 11.05.2021, 11:38
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,528
Default

Quote:
Originally Posted by Wheezer View Post
The only quick it that one link group may not be renamed. NO big deal, looking at the "Downloaded from" column indicates that all the links are the same source URL, so just a simple single rename.
I didn't quite understand. Can you give an example?
Reply With Quote
  #27  
Old 11.05.2021, 12:07
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

I will run several more batches tonight and write up examples for you.
It seems like there is about a 99.95% success factor in the package splits.

So far out of about 8000+ links processed, I have seen four failure cases.
In two(2) cases the links remained in the original package, but those were all for the same source URL. In one case a package was renamed that had contents from two source URLs.
However I just fixed them manually.

For the next several batches I will save the failure states and write up details on them. I saw a bit of similarity, but want to be exact sure before I let you know.

More when I process several more sets.

Cheers!
... W ...

Last edited by Wheezer; 11.05.2021 at 12:13.
Reply With Quote
  #28  
Old 11.05.2021, 17:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

@Wheezer: Could you please try to open the Eventscripter Editor and after it failing to open, create a log? see https://support.jdownloader.org/Know...d-session-logs and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 11.05.2021, 20:28
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

@Jiaz: I solved the problem, it no longer occurs. I've diagnosed it to "pilot error" not aircraft fault.

After adding the script manually (thanks to mgpai) the editor came up as expected and works. Prior to that I was experimenting to see how that section worked. In the process I added one, deleted one and disabled one. Basic monkey around to see how it worked. In the process I may have removed all of them (or the default or sample script). After that the editor would not come up. Yet this was resolved when I manually injected the script.

I don't know your code internals, but it appears that there might be an assumption made of a default script to start the editor. Lacking that the editor would not start. I've seen similar situations in code for other editors.

When package testing is done I can load a clean JD and try to recreate the steps to make it fail again and then send logs if it would help.

Cheers!
... W ...
Reply With Quote
  #30  
Old 11.05.2021, 21:12
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,044
Default

Thanks for your detailed feedback and yes indeed knowing how to be able to reproduce this issue would be very helpful!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #31  
Old 11.05.2021, 21:33
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

No problem ... will generate editor tests info for you once the testing the package script testing is done. It has been interesting.

It seems that a background process occasionally launches that pauses processing and results in the script acting odd. One or two links from a preceding package are processed, resulting in a spurious odd package. Almost as if the ID for the link was mixed up.

Not sure if a background SAVE is occurring during processing that is causing it, or just the size of the packages being processed (20K to 50K links per package). At this point I am just running the script on packages to capture data so I can make a coherent description of the problem.

Cheers!
... W ...
Reply With Quote
  #32  
Old 12.05.2021, 06:26
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

@mgpai - I finally for all the links processes, here is some information.

The packagizer script worked well, however there were some odd actions. In no cases were links dropped, rather it some cases is seems like the script took links from a preceding packager and used those to make package. This happened several times when the script paused due to a background JD process. It seems to indicate that the pointer to the link changed, so the script processed the wrong link. No major problem, those are easy to identify and correct.

For the test a total of 18 packages were processed, 6 processed with no problems, and 12 had odd links from a preceding package added. The total processed was 413257 links, with only 67 links being processed incorrectly, or a 99.9998% success rate.

I looked at the size of the package and that does not seem to have a repeatable impact.
The following are the package sizes and processing results:

Package - source links - processed links - results
(R) - 9927 / 9927 - All links processed correctly
(Q) - 1032 / 1032 - All links processed correctly
(P) - 37440 / 37446 - 6 links added from preceding package
(N) - 12142 / 12149 - 7 links added from preceding package
(M) - 56172 / 56190 - 18 links added from preceding package
(L) - 16098 / 16102 - 4 links added from preceding package
(K) - 13886 / 13888 - 2 links added from preceding package
(J) - 9453 / 9455 - 2 links added from preceding package
(I) - 5453 / 5453 - All links processed correctly
(H) - 32295 / 32303 - 8 links added from preceding package
(G) - 22797 / 22804 - 7 links added from preceding package
(F) - 27685 / 27886 - 1 links added from preceding package
(E) - 15392 / 15402 - 10 links added from preceding package
(D) - 18402 / 18403 - 1 links added from preceding package
(C) - 55415 / 55416 - 1 links added from preceding package
(B) - 45716 / 45716 - All links processed correctly
(A) - 30717 / 30717 - All links processed correctly
(0) - 19268 / 19268 - All links processed correctly

So for most cases the script would run flawlessley. I think in my case the large number of links in the download list and linkgrabber list caused the problems seen. From what I can see it is either JD or a JAVA problem.

Many thanks for the utility!

Cheers!
... W ...
Reply With Quote
  #33  
Old 12.05.2021, 06:44
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

Update: I think I found the answer, and it seems quite simple.

The majority of the links had TWO source URLS, one for the CONTAINER URL and a second for DATA URL. These all processed fine.

However some direct image links got into the lists, these only have a DATA URL.
When the script attempted to process those with a single URL, that resulted in the odd package results.

Plus the safest way to process is to have the packages at the TOP of the link grabber list. Thus if a bad file gets mixed it does not attempt to grab links for the packages above it.

I only noticed it when I was cleaning up the odd package results.

Cheers!
... W ...

Last edited by Wheezer; 12.05.2021 at 07:05.
Reply With Quote
  #34  
Old 12.05.2021, 07:25
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,528
Default

Quote:
Originally Posted by Wheezer View Post
However some direct image links got into the lists, these only have a DATA URL.
When the script attempted to process those with a single URL, that resulted in the odd package results.
The script processes only those links which have container url. Links without one will be ignored, and hence will remain in the existing package.

It can be modified to use content (DATA) url as fallback, if the attributes are similar to the container url. Else, while they will be moved to a new package, they woiuld still end up in individual packages.
Reply With Quote
  #35  
Old 12.05.2021, 07:53
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

Hmmm ... While it SHOULD have stayed in the source package, upon execution several made it through. None remained cleanly in the source package, all got moved. Plus causing the injection of odd links from the preceding package.

However, it seems that for each single URL file found, it caused two links to be moved to a package that did not correspond with the appropriate name. Odd but now it seems to match a pattern.

Yet only 67 glitches out of 413257 links .... they don't do that well with brain surgery

I'm happy, great script.
... W ...
Reply With Quote
  #36  
Old 12.05.2021, 11:10
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,528
Default

@Wheezer: The script processes the selection as a set of links and does not take into account the existing packages at all. Since the script generates the package based solely on container url, you should not find one package containing multiple container urls, or multiple packages for a single (same) container url.

Theoretically, if all 413257 links were in a single package and you run the script, it should result in one package per container url. Links which do not have container urls should remain in the package which originally contained those 413257 links.

If you come across any cases where it behaves differently, please provide me the example links. I would definitely be interested in looking into it.
Reply With Quote
  #37  
Old 12.05.2021, 12:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

@Wheezer: With that many links in list, it is important that Java has enough memory to work with.
unfortunately I'm not able to reproduce the editor issue, so would be good if you could try to reproduce the issue and create a log, see https://support.jdownloader.org/Know...d-session-logs
and post shown logID here. Then I can also check about memory constraints
__________________
JD-Dev & Server-Admin
Reply With Quote
  #38  
Old 12.05.2021, 12:44
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

@Jiaz: I will backup my configuration and clear the scripts to recreate the editor situation. Memory should not be a problem. My download system has 96GB and over 56GB free and available with JD and everything else up and running. Disk I/O for JD files and system temp and such is hosted on NVMe SSDs, so there should be no disk delays.

Once I recreate the failure mode I will write up the steps I took and generate a log.

Thanks!
... W ...
Reply With Quote
  #39  
Old 12.05.2021, 12:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

@Wheezer: Check about dialog of JDownloader, see https://support.jdownloader.org/Know...tion-directory
it states the java memory settings. Can you provide screenshot?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #40  
Old 12.05.2021, 13:00
Wheezer Wheezer is offline
JD Addict
 
Join Date: Mar 2009
Location: WheezerLand
Posts: 158
Default

Quote:
Originally Posted by mgpai View Post
@Wheezer: The script processes the selection as a set of links and does not take into account the existing packages at all. Since the script generates the package based solely on container url, you should not find one package containing multiple container urls, or multiple packages for a single (same) container url.

Theoretically, if all 413257 links were in a single package and you run the script, it should result in one package per container url. Links which do not have container urls should remain in the package which originally contained those 413257 links.

If you come across any cases where it behaves differently, please provide me the example links. I would definitely be interested in looking into it.
That too is what I thought when I saw the code, a straight forward process.

However in practice there were times that it did not work that way. There WERE multiple URLs in a package, and I could see the package above the target decrease in links when the odd items occurred. I confirmed it since the packages were all alpha sorted. Thus package (C) originally had no links starting with "B". Then total links shown for (B) decreased by one, and a "B" named link showed up in the processed results.

This led me to consider if the internal pointer that refers to the actual link information may have been an issue. Thus when the movetoNewPackage runs it (for example) requests to move item id 0003, but internally the system supplies information for item id 0008. It "works" as planned, but just on data that was supplied incorrectly, at a lower level it would not comprehend.

I manually cleaned up the odd links, so I will now have to recreate the fault situation and send you the sample links.

Cheers!
... W ...
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 04:50.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.