#3201
|
||||
|
||||
@Neurotoxin001: thanks for the report. will be fixed with next update
__________________
JD-Dev & Server-Admin |
#3202
|
|||
|
|||
Packagename - Current problem.
The important character "_" disappears Example: 12345 12345 Shouldbe: 12345_12345 Example link decrypt: **External links are only visible to Support Staff****External links are only visible to Support Staff** |
#3203
|
||||
|
||||
@Student im ersten Jahr: disable Settings->Advanced Settings->GeneralSettings.cleanupfilenames
__________________
JD-Dev & Server-Admin |
#3204
|
||||
|
||||
VK.com problem: Analysis
VK.com plugin is working again, but when I add e.g. 100 video links to LinkGrabber, only 20 to 30 % will be found. I can add those links once again and let LinkGrabber grab again, a few more links are found. Have to repeat this several times until all links are found/grabbed. Reason is: VK.com introduced a change which will refuse answering too many requests in a short period of time. I can use Extreme Picture Finder, configure a random delay, between the requests/downloads, and everything works fine. To solve the problem in JD, also here a delay has to be introduced. This is the only solution I found so far. |
#3205
|
||||
|
||||
@StefanM: Please don't open multiple threads with same topic!
JDownloader already has a default wait of 1 secs for pagination/crawling (see Settings->Plugins->vk.com) Please provide a debug log, see https://support.jdownloader.org/Know...d-session-logs Enable Debug first, see Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader. Now wait for issue to happen and then create log and post shown logID here vk.com now has a WAF (web application firewall) that we added support for recently and I could not reproduce any issues. So please provide a log, so we can see why it fails for you
__________________
JD-Dev & Server-Admin |
#3206
|
||||
|
||||
[vk_com] how to set download limits for specific files ?
Hello.
I have few galleries on my list. Unfortunately with number of download files (that I can get in the same time) set for instance to 5, VK says immediately that I want files to be downloaded too quickly. That the time is too short. I wonder if there is a setting, that can make JD download files with any download transfer limit or sth ? But... only for galleries, image galleries. Videos must be downloaded without such settings. The thing is videos have no such errors while downloading. Only images. These galleries have 3k images, 4k images.. but some have 860, or just 430.. the same error occurs with all of them. Only with these images, not videos. Link (gallery) example: **External links are only visible to Support Staff****External links are only visible to Support Staff** Last edited by elvis_ef; 30.05.2022 at 22:07. Reason: typos |
#3207
|
||||
|
||||
Merged vk threads.
@elvis_ef We got one generic vk.com thread. Please do not open new threads for vk.com related issues. To answer your question: No there is no general limits you can apply. We will simply have to update the plugin in order to recognize that limit so it can wait- and try again later. Please post your log-ID here | bitte poste deine Log-ID hier. -psp- EDIT Auto-retry-handling for "Too many requests" has already been implemented. Currently it will wait 60 seconds for all vk.com URLs and continue afterwards...
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 30.05.2022 at 23:08. |
#3208
|
||||
|
||||
Vk.com Vkontakte.ru - not 'finding' all links in a single run
Quote:
Will conduct a few tests and then send you results and more details later this week. |
#3209
|
||||
|
||||
VK.com - Test documentation on new issue
Quote:
Please find a detailed test documentation attached as zip archive. I invested some 90 minutes for that documentation. Hope, it helps solving the issue. Excerpt as plain text: Test documentation Notes: I'm logged in with my VK.com credentials during this test. The behavior shown here is typical. Had dozens of cases such as this one - on two different machines using two different Internet providers. Both of them were 16 Mbps accounts. How the test was run: (I'm quite convinced, you will be able to reproduce this) First, I copied all 2,166 video links from this example page: **External links are only visible to Support Staff****External links are only visible to Support Staff** - grabbed with LinkGopher in Firefox. You will find all 2,166 links in Sheet 1 of the attached Excel book. In sheet 2 you will find all links found by LinkGrabber in the first run. For the next runs, each time I copied all 2,166 links again to LinkGrabber. I used the order as in the Excel book. In sheet 3 you will find all additional links found by LinkGrabber in the second run - i.e. only those links, LinkGrabber did not find in the first run. Same applies to the third run… 1st run: 863 found 2nd run: another 468 found 3rd run: another 310 found 4th run: another 210 found total 1,851 links found after 4 runs, would need a few more to find all online links. In plugin settings, I left those 'sleep time' settings alone, as default that is. But I noticed, that there is no 1 second delay between grabbing individual video links. As you can see from the attached bubble screenshots (time taken and number of online links found), more than just one link is grabbed within a second. E.g. 973 links found online in 13:34 (814 seconds). This is just an average value. I often see more than 3 or 4 links added as online within one single second. Maybe I misunderstand that setting? If yes, can you please explain what this 'sleep time' is about. Last edited by StefanM; 31.05.2022 at 17:15. Reason: Added Title |
#3210
|
||||
|
||||
Nothing to thank me for. And please understand and take it personal that I did delete your other post. Just please don't open multiple threads/posts about the same topic. We see your post
@StefanM: It would be nice if you could just provide a debug log. Please understand that it's much FASTER and easier to have read to analyze logs that show the issue than us having to reproduce the issue and wait for it to happen. So instead of spending 90 minutes in documentation, just reproduce the issue once (crawler abort) and then create the log. Also you should not crawl/find the individual links by yourself because adding 2166 links will result in minimum 2166 requests (likely even more) . You should instead just let JDownloader crawl/find the links as it can find multiiple links with less requests. Crawler in JDownloader will be much faster than adding thousands of individual links by yourself where no optimization (api with multiple results) can take place. Quote:
vk.com has a generic request limit of 500 ms between each request and max 15 requests every 30 secs Quote:
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 31.05.2022 at 17:53. |
#3211
|
||||
|
||||
Another explanation for the wording "pagination" and our related setting "Sleep time for general pagination":
Pagination is when a website either contains multiple items per visible page or e.g. uses "endless scrolling" like vk.com does so every time you scroll down more, X new items get loaded. Our setting defines the wait-time in between. By using "Link gopher" to manually collect all single video items, the following things will happen:
Is there any good reason why you do not let JD crawl this "wall" link? Code:
vk.com/video/@CENSORED Letting JD do the job you now did by hand using "Link Gopher" should make things much faster/easier. Nevertheless I'm currently adding all of these 2166 items from your excel sheet to my test-JD and I will post my results here afterwards. -psp- EDIT Here are my results: Packages: 1676 Links: 1730 Links online: 1679 EDIT2 Here are my results when I let our crawler do the job (with pagination etc...): Packages: 6 (much less packages because all vk.com self-hosted videos of that profile will be grouped intop one package named after the username) Links: 2527 Links online: 2466
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 31.05.2022 at 18:12. |
#3212
|
||||
|
||||
Quote:
But couldn't you just add those links I sent you to JD's LinkGrabber and see for yourself what happens? Opening Excel, copying the links from sheet 1, and adding them to LinkGrabber wouldn't take more than a minute. And then you can see for yourself what happens. I add those video links as single links for simple reasons: 1. In JD, I can only see the individual links in my download list, wenn I add them separately. 2. I can only have the link as part of the downloaded video file name, when I use individual links. 3. In most cases I don't want to download all videos. Here it was just a test. 4. And maybe later - e.g. after more videos have been uploaded - again I only want to download a few of them. Now, once again my little request: Before I create a log with sensitive data (maybe alleged copy right infringements, ...) could you please just add those links to your JD and see if you can reproduce the problem? It won't take you more than a minute and then waiting what happens. I would really appreciate this! I can tell you that the problem did not exist before the VK.com change. And I can tell you, that this problem does not exist, when I use Extreme Picture Finder (**External links are only visible to Support Staff****External links are only visible to Support Staff**). I have a working template which worked before and after the changes VK.com made. This template also selects the highest resolution only, and the log provides me with the direct links, which are valid for some 5 days. No problem occurs, when I feed those links to LinkGrabber. Just the file naming won't work then. Addendum: When I let JD crawl the video page, it stops crawling after 1,001 links. Last edited by StefanM; 31.05.2022 at 19:33. Reason: Addendum |
#3213
|
||||
|
||||
@StefanM: I can reproduce the issue but I did not ran into any plugin issue. For me it stopped at about 1900 found links. The issue must lie somewhere between. That means either some links are getting *lost* from clipboard to plugin or there are some error messages in plugin that are silently caught.
We will do some more testing by tomorrow Quote:
pspzockerscene and I already did hence my comments above
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 31.05.2022 at 19:56. |
#3214
|
||||
|
||||
Quote:
Just out of interest, did you also do testing with the account disabled in JDownloader?
__________________
JD-Dev & Server-Admin |
#3215
|
||||
|
||||
Quote:
And If you look at the different runs in my Excel book, you can see that each run some other links are found. But everytime, each run, all links are in the LinkGrabber window. |
#3216
|
||||
|
||||
Quote:
And I/we should not not mix up two issues: 1. Pasting individual video links into the LinkGrabber window. (many links are not found, more are found in a second, third, ... run) 2. Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found) Last edited by StefanM; 31.05.2022 at 23:17. Reason: typo and more |
#3217
|
||||
|
||||
Quote:
Will answer all questions in a new post later today. Just one hint right now. I learned from you that VK.com only accepts 15 requests within a 30 sec interval. For sure, my test with copying more than 2,000 links to LinkGrabber sent more requests, than 15 per 30 sec. But I will collect all details now and give you a summary including some analysis later today. Thanks again for taking my report seriously. :-) |
#3218
|
|||||||
|
|||||||
Quote:
https://support.jdownloader.org/Know...iew-added-urls Quote:
Please add a more detailed explanation. Quote:
Quote:
Quote:
It worked fine here. Please simply provide a log. We do not process your data in any way. If you want you can also manually grab your logs from the "logs" folder inside your JD folder, zip them and upload them to a cloud of your choice and delete them once we got them. Quote:
Afaik vk definitely allows more requests but our plugin limits them atm. Quote:
Please provide a log... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3219
|
||||
|
||||
More Findings and Test Results...
Quote:
I'm referring to two different issues: Issue 1: Pasting individual video links into the LinkGrabber window. (many links are not found, more are found in a second, third, fourth, ... run) Issue 2: Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found) A) Logged in / not logged in on VK.com (tested with issue 2) Tested several times on two different machines with two different providers. Yes, there is a reproducible difference: When I'm logged in, crawling is much slower and in average only some 50 % of the links are found. When I'm not logged in, crawling is about twice as fast and in average over 90 % of the links are found. B) Different providers (tested with issue 2) Even though provider speed tests show almost the same speed, crawling speed is reproducible different. C) Your info: VK.com only accepts max. 15 request in 30 sec (tested with issue 1) While crawling (Finding Links…) I see a number of offline links being reported. This means to me that already in this stage requests are sent to VK.com. E.g. after 3 minutes 547 online and 32 offline links are displayed in the bubble. Isn't this already 579 requests within 180 seconds? This is for sure above those 15 accepted requests per 30 sec interval. Can this be the reason for issue 1. Please have a look at the attached Excel book which might give you an explanation. I marked all files by colors, in which run they were found. So you can see when which links were skipped / not found. D) Browser.setBurstRequestIntervalLimitGlobal("vk.com", 500, 15, 30000) Is there any way for me to edit those numbers for conducting more tests? Cannot find it in advanced settings. E) Why do I prefer to copy individual links to LinkGrabber? It is because of a limitation in JD :-)
F) You mentioned reducing number of requests by dupe checking As pointed out in C) there seem to be many occasions, where requests are being sent in high frequency. Please correct me if I'm wrong. What I would like to know is, how and when dupe checking is being performed. E.1) Let's assume, I copy links to files to LinkGrabber, which already had been downloaded earlier. Does JD dupe checker first check DownloadList and LinkCollector zip-files, whether or not they contain one or more of the added links? Or does JD send requests anyway? E.2) Let's assume I add 20 links to LinkGrabber and links 11…20 are dupes of links 1…10. Now let's assume, while crawling for links 1…10, JD only finds links 1…5. links 6…10 are not found. Now, what happens with links 11…20? Will JD know that it already found links 11…15? And will JD try to find links 16…20 once again as it had not found links 6…10. Again, please note that 16…20 are identical with 6…10. G) Dupe checker doesn't work in a consistent manner I'm mentioning this here, because you were referring to dupe checker. What 'annoys' me is the fact, that - depending on where the dupes are - and what I did with them, they can be displayed or they cannot be displayed (checking the box 'Already in Downloadlist'). Links that were already downloaded won't necessarily show up in the LinkGrabber pane. So, the user won't know, if JD simply 'skipped' those links due to an error or because they already had been downloaded earlier. Or is there a way to have them displayed as 'already downloaded earlier'? And one more question regarding bug reports or requests for improvement: To give you just one more example: The bubble notifier position settings (in advanced settings) do not work correctly when I use a display scaling different from 100 % - tested on different machines. Where should/would I report this? In fact, over the years I found quite a few issues, but never took the time to report them here. Maybe I should do that… :-) Thanks again for any efforts taken. You guys are really doing a great job! Last edited by StefanM; 01.06.2022 at 18:35. Reason: typos, answers added |
#3220
|
||||
|
||||
Quote:
More to come within the next hour. |
#3221
|
||||
|
||||
Quote:
I see different logs such as: vk.com_jd.plugins.decrypter.VKontakteRu.log.0 8.648.264 bytes vk.com_jd.plugins.decrypter.VKontakteRu.log.1 11.201.511 bytes vk.com_jd.plugins.decrypter.VKontakteRu.log.2 11.131.088 bytes vk.com_jd.plugins.hoster.VKontakteRuHoster.log.0 vk.com_jd.plugins.hoster.VKontakteRuHoster.log.0.lck vk.com_jd.plugins.hoster.VKontakteRuHoster.log.1 vk.com_jd.plugins.hoster.VKontakteRuHoster.log.2 vk.com_jd.plugins.hoster.VKontakteRuHoster.log.3 vk.com_jd.plugins.hoster.VKontakteRuHoster.log.4 .... Tons of data... :-) Before I upload anything, I want to be able to verify what I'm sending. And limiting the amount of data would save me a lot of time. PS: Worked for an IT company, would even call myself an IT-specialist, and I'm also a computer forensics guy,... So, please understand that I may be overcautious. PPS: I'm a good friend of the EPF developer, Maxim from Ukraine. That's why I'm pretty familiar with EPF and also because I'm writing the German GUI for all his programs, same as some of his web pages :-) |
#3222
|
|||||||||||||||
|
|||||||||||||||
Both understood.
Can't confirm. Yap. I haven't denied that. Quote:
Quote:
You can grab our source code and edit those as explained here: https://support.jdownloader.org/Know...up-ide-eclipse Our currently set limits are not linear. Here is a quote of the comment of that line of code we referenced: Quote:
Quote:
The mentioned support article describes what you need to do in order to see the individual links even when adding content via "vk.com/video/@CENSORED". This is working fine here. Quote:
https://support.jdownloader.org/Know...the-packagizer If you still fail to do so, please provide detailed examples (URLs!) so we can help you. Quote:
While we will look into this I just want to point out that e.g. on github.com you will find specific scripts dedicated just for downloading from vk.com. You might be better off using them in this case... Quote:
Quote:
Quote:
In this case, links first go through a crawler to find the video direct-links/external links. All of this will always be done and causes http requests. There is no way to avoid this other than maybe using EventScripter scripts especially written for this: https://support.jdownloader.org/Know...event-scripter Quote:
Quote:
First of all, JD only checks against items that are still in your downloadlist. If you e.g. remove them from your downloadlist but the files themselves still exist in your download destination, JD will not detect those as dupes. This makes sense because of multiple factors that I won't explain right now. A user once wrote an EventScripter script for enhanced dupe checking. That might be helpful for you too: https://board.jdownloader.org/showthread.php?t=83048 Quote:
https://board.jdownloader.org/forumdisplay.php?f=12 Quote:
...but please save yourself and us a lot of time by simply providing logs: - You don't have to analyze JDs behavior from the outside as it won't necessarily help finding the cause of that issue - We can work on it more efficiently by having logs Quote:
Code:
vk.com_jd.plugins.decrypter.VKontakteRu Quote:
Great! If you want you can jump right in and help finding the issue
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3223
|
||||
|
||||
Quote:
But I will take a look at it... Quote:
It works after I unchecked everything, but 'the address to the actual content' However, the wording is not correct. It is the address I want to have, so I'm happy with that. But the address to the actual content is something like this: **External links are only visible to Support Staff****External links are only visible to Support Staff** Quote:
PS: Being already retired, indeed I would be prepared to help (if I could), which I've already been doing voluntarily for a number of software developers in the past years. But again, I'm not a software developer, just a retired IT guy... ;-) Last edited by StefanM; 01.06.2022 at 19:17. |
#3224
|
||||
|
||||
Things can always change
Quote:
Quote:
In a lot of cases it is impossible to direct-link content e.g. because - Direct-URL can only be used once - Direct-URL expires after a short time - Direct-URL is only valid for current IP address - Content is hidden behind a POST-request and not a GET request -> In this case it would be impossible for the user to use the, by your definition, "content-url" --> This means by our definition the current "content-URL" is an URL which, if you open it, links either directly or indirectly (as closest as possible, most times 'indirectly') to the content behind. E.g. for an instagram post containing multiple pictures, the cuntent-URL of any of those pictures would simply open the main post as it is not possible to "officially" link any image of a multi-image-post. It's similar for twitter and we're handling it the same way for vk.com. Also the content-URL shall not be a temporary URL because again, if the user tries to use that one and it doesn't work anymore -> Problem. Ultimately at this moment it's up to the plugin developer which URL to choose as content-URL. EDIT So if you got a better wording than "content URL", let us know Quote:
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 01.06.2022 at 20:34. |
#3225
|
||||
|
||||
Quote:
__________________
JD-Dev & Server-Admin |
#3226
|
||||
|
||||
Results of my investigation so far:
The issues were caused by a mix of different uncaught errors some of which were simple website-errormessages upon which a page reload is required: - "You tried to load the same page more than once in one second" -> While afaik we didn't do that, this message can happen for about 1% of all videos. I've implemented up to 10 retries in this case but it doesn't yet work as it should - In some very rare cases the regular expression used to filter the json containing the video information ws not precise enough and failed to do so - No matter what Jiaz will find out the issue was not caused by our eventually failing request-limiting as I've removed that completely for my tests - Another unhandled error (but that only affected one of your 2166 items): "This content is blocked in your country" It's still not working as it should but here are my current results when adding all of your 2166 links without account: Packages: 2525 Links: 2592 Online: 2528 Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3227
|
||||
|
||||
If needed, I made a couple of screenshots showing this. It happened in every test run I made today.
|
#3228
|
||||
|
||||
Addendum
And it's easy to see, that the 15 requests per 30 sec doesn't work:
Will now setup a clean JD2installation to produce the logs you requested |
#3229
|
||||
|
||||
After a huge refactoring of our vk.com plugin, the "You tried to load the same page more than once in one second" error should be successfully workarounded with the release of our next set of CORE-updates so all of your 2000+ vk URLs should get processed just fine.
Keep in mind that once the mentioned error appears once, all crawler instances will wait for it to be cleared -> vk.com crawl process may seem slower than usual but therefore it should be able to process all items. Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3230
|
||||
|
||||
Quote:
__________________
JD-Dev & Server-Admin |
#3231
|
||||
|
||||
Quote:
I set up a clean JD installation using JDownloader2Setup_windows-x64_jre8.exe 'offline installer'. Notes:
I started my tests using those 2,166 separate links from the Excel book I sent you earlier. I did NOT use my VK.com credentials. Result: 2,021 Packages 2,525 Links 2,459 Online 63 Offline Hosters 3 Unknown Status 3 Time taken: less than 10 minutes Silly me: I forgot to copy the log files before re-starting with a new clean program folder As a next step I applied all my settings (quite a few!) of my other JD installations to see if that would make a difference. Result: 1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>) 2,609 Links 2,538 Online 0 Offline Hosters 3 Unknown Status 71 Time taken: less than 10 minutes Finally, I conducted the same test with my 'Work Version' of JD Notes: This version has 150,000+ files in the downloads pane - most of them downloaded in the past years. This time I was logged in to VK.com Result: 1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>) 2,605 Links 2,535 Online 0 Offline Hosters 3 Unknown Status 70 Time taken: 41 minutes Started the same LinkGrabbing once again, but NOT logged in. Result: 1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>) 2,607 Links 2,537 Online 66 Offline Hosters 3 Unknown Status 4 Time taken: 20 minutes As you can see,
I assume that speed is still slower than with a clean installation, as JD needs some time for dupe checking within 150,000+ links. Is my assumption correct? Could add more results later, but what I can say already now is, that those severe problems (finding less than 50 % of links copied to LinkGrabber) seem to be gone. Just let me know which scenarios I should test. Question: Did you release any updates in the meantime? If yes, I should repeat my test some other day and disable updating then. However, I better wait until you're done with your updates of the VK.com plugin. Last edited by StefanM; 02.06.2022 at 18:45. Reason: Added one more set of test results |
#3232
|
||||
|
||||
Quote:
Please use forum search on this. Short: We update JDownloader VERY often (10-20-30 updates a day can be normal) that an offline installer would be outdated a few minutes/hours after its creation. Also the installer only installs files that are required (eg OS, CPU architecture, bitness...), so we would need multiple offline installer. Wasted time of work/traffic and ppl just will hotlink/spread an outdated installer. Our installers are webinstallers that always install the latest available version.
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 02.06.2022 at 18:36. |
#3233
|
||||
|
||||
Quote:
Also even though Jiaz stated that we do not need any new logs right now: In the future please rather include logs than doing some hour long tests for yourself: It doesn't help us... -psp- EDIT Also, I did all of my tests without vk.com account. As stated the problems were not account related.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 02.06.2022 at 18:36. |
#3234
|
||||
|
||||
Quote:
__________________
JD-Dev & Server-Admin |
#3235
|
||||
|
||||
Quote:
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3236
|
||||
|
||||
@StefanM: new set of updates are live
__________________
JD-Dev & Server-Admin |
#3237
|
||||
|
||||
Quote:
many links might need memory tweak/optimizations. but 150k is no problem with good memory settings, even million and more
__________________
JD-Dev & Server-Admin |
#3238
|
||||
|
||||
Quote:
All possible combinations of those are at least 4 different tests. And - as you can see - you need to repeat tests several times to achieve a representative result - for example to make sure that no other factors may have influenced the result. For instance caused by VK.com. -psp- Quote:
You can also observe another difference: Whether or not files are being reported as offline by JD depends on whether or not I am logged in. One reason might be, that in some video galleries you do not have access to all videos when you are not logged in. According to my experience, these restrictions not only apply to VK.com but to many other websites as well! And again, speed has an influence whether or not a timing related problem shows up. And according to my observation it also makes a big(!) difference when JD has to scan a large amount of already downloaded files. In my case this led to a tremendous crawling time increase - reproducible! |
#3239
|
||||
|
||||
Quote:
For example the initial report of yours -> start Jdownloader -> add the links -> wait for crawler to stop -> less links than insert = issue, create log -> that's all
__________________
JD-Dev & Server-Admin |
#3240
|
||||
|
||||
Quote:
What has massive influence on speed is this, see https://board.jdownloader.org/showpo...postcount=3237 Most influence is caused by rate limits/server response/latency and what kind of links you're adding (folder links with 1 requests = 1000 results vs 1000 single links which require 1000 requests)
__________________
JD-Dev & Server-Admin |
Thread Tools | |
Display Modes | |
|
|