[dominika992]

Hi

I have always experienced this issue but this time I am trying to crawl something that has many, too many of this type of content.

When some items are being crawled it stops at some point, some % and its then like begginng, 0 kbps, starting or 493kb/s (for e.g.) and not moving forward, not progressing.

Maybe the data/content is broken somehow?

But the JD tries to downlaod it and download and download... and never fail. The consequence is that I can't download the whole thing, because if I have 8 connections and JD tries to crawl 8 links like that at the same time that it downloads for infinite amout of time and not moving forward it never ends...

Then only works by me coming in and manually deleting this few links, and it then works and downloads many many contents until it again gets fixed on broken links and not progressing...

Here is the video showing the issue on few links:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

And here are few links I have this problem with:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Even in the browser they appear broken somehow, if png/jpg its like trimmed, cut, can't see the whole. And if .gif typically it stomps at some point and not moving forward.

I have only here a gifs examples, I already deleted some jpg links.

I think if JD can't download it cuz broken or smth then it should skip this files and mark as OK but broken or something?
pls help

[Jambalaya]

I have noticed this (only) with bdsmlr as well, but haven't logged a ticket yet. It seems to be related to larger files of about 3.8MB and up.

[pspzockerscene]

Please wait until other team-members find the time to investigate this.
Your thread hasn't been forgotten!

[dominika992]

I can relate it probably is that way with bigger files, mostly (or only?) gifs.
It is very possible.

[pspzockerscene]

Originally Posted by pspzockerscene

Please wait until other team-members find the time to investigate this.

...

[Jambalaya]

I thought about trying to download some of these larger files from outside JDownloader and was investigating what I would need to do. I believe the issue is with the files themselves, not JDownloader. Using developer mode in Opera, this is what I found for one sample file:

GET **External links are only visible to Support Staff****External links are only visible to Support Staff**
net::ERR_HTTP2_PROTOCOL_ERROR 206

Trying to save the file manually from the browser results in 'network error'. It might be the best JDownloader can do for these is to fail them and move on to the next file in the queue rather than to keep trying them infinitely.

[dominika992]

Maybe I have found another issue.
Like this link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
This is a post that is available.
You can see it in the browser + the picture of this post + download it etc.
JD recognizes this post but can't download it.

An errror occured!
File not found.

But it is there

[pspzockerscene]

Originally Posted by Jambalaya

Trying to save the file manually from the browser results in 'network error'. It might be the best JDownloader can do for these is to fail them and move on to the next file in the queue rather than to keep trying them infinitely.

Yap and we'Ve also investigated this problem some time ago but I forgot to answer - I'm sorry.
The problem is that those files are incomplete on the server.
For .gif files you can't actialy see it as they will just stop in the middle but for pictures (I'm lacking examples atm.) you can actually see that the picture starts to load and all of the sudden, the rest/half of the picture is displayed grey.

This is how I remember, my OS displayed pictures after a HDD crash a long time ago.
I recommend contacting the bdsmlr.com support about those links and ask them to try to fix it.

Originally Posted by dominika992

An errror occured!
File not found.

But it is there

This problem will be fixed with the next set of CORE-updates.
You will need to delete the old findings and re-add such posts.
In that specific case, the new crawler version will find two version of that image one which is online and one which is offline:
Offline ends with: ZCDcWYDboD-og.jpg
Online ends with: ZCDcWYDboD.jpg
EDIT
If you open the html code of the file, you will find the broken image-url besides this html snippet:

<a class="magnify" href="

... and the working one here:

<img class="sidepostimage"

Sure you could also correct such links manually until we release the next set of updates.
This would e.g. be possible using a LinkCrawler rule of type REWRITE.

[Jambalaya]

In case it helps anyone, it seems the problem on the bdsmlr side happened sometime in 2021-01, and was resolved just prior to 2021-08. Since all their files are dated, you can work around the broken range fairly easily.

Up to the end of 2020, and from 2021-08-01 onward seem fine. Within the broken range, files larger than about 3.9MB are very likely incomplete; there is the odd good one in there, but not worth searching for.

[pspzockerscene]

I know I'm repeating myself but I'd report this issue to the bdsmlr.com support and kindly ask them to fix it. That is of course only possible if they still got the original un-damaged files.