[djmakinera]

Interesting :
JD 2 - 30000 photos = 20 hours (if finish parsing error will crash or not)

Tested:
another application - 30000 photos = 1 hours 30 min

[raztoki]

on your Original link the album is only 269 images, Took me 2.5minutes on my new code. The only way to make it faster again is to remove half decrypter code and make it into a hoster plugin. Probably the reason why the other program is so much faster, as the adding of links is only small component of the task. If online checking and dl components done its own plugin, even at the download stage you could increase the efficiency of the plugin by about 10 fold, siding scale depending on the size of the gallery. The decrypter does everything one step at a time, and this can effect over all time lapsed assuming large tasks.

[djmakinera]

Raztoki - Is JD - first scans page 1, page2, Page3, Page4 etc. ? and then analyzes and then processes the links?

[raztoki]

yah, not sure why we don't use the all image page (/main/pic_tape.php?ad=uid), think its always the biggest image, we would just need to goto x pages vs, x * y to get finallinks. I'll ask psp later, maybe they do not show all images/pic tape for all albums?

[djmakinera]

Do not use (/main/pic_tape.php?ad=uid)
because if the image is large in size (full size) - does not show the maximum size

**External links are only visible to Support Staff****External links are only visible to Support Staff** (NORMAL SIZE) (/main/pic_tape.php?ad=uid)

**External links are only visible to Support Staff****External links are only visible to Support Staff** (FULL SIZE)

[raztoki]

I compared the link returned from the current implementation in JD and then went to pic_tape to find the same image and it was the exact same image url. So not sure what you did but I can't reproduce your findings.

If we use this method it will return all links as you only need to hit each 'x pic tape page' (* 12 a page) = total images vs 'each album page y * (48 -1) each image page = total images', in the standard album layout.

[djmakinera]

Example:

_**External links are only visible to Support Staff**Red">b0.eu.IMGSRC (NORMAL)

_**External links are only visible to Support Staff**Red">o0.eu.IMGSRC (FULL)

[raztoki]

I saw that, and that was not my findings.

[djmakinera]

**External links are only visible to Support Staff****External links are only visible to Support Staff**

logfile:

http://jdownloader.org/pastebin/110969

[raztoki]

Seems it only happens for new links, and I made support based from the html in your log. That said I was able to find some new links on the site also to test it with, and it works. In the process of rewriting the plugin to speed up efficiency. Just need to find some passworded links so I can implement/test the new code!

[djmakinera]

raztoki
Still loses to time hehe

Test:

JD (12 photos)
31 second
47 second

Other program (12 photos)
3 second
4 second

[pspzockerscene]

JD is no site crawler/ripper, at least it's not made exactly for that.

GreeZ pspzockerscene

[djmakinera]

I know, but it was addressed to raztoki. We have a common theme hehe

[djmakinera]

"Broken parsing" (probably all links)

I do not know what happened but JD2 shows that all of the links "out of date"
Example:

Vk.com - broken
Imgsrc.ru - broken
zippyshare.com - broken
etc...



I tested different installations of JD2 (without updating!) - All the links work.
So what happened?

remove hosts and works all

[raztoki]

not sure why I didn't post here

so answer for everyone else is..

Originally Posted by raztoki

try closing jd
goto install folder
delete that file jdownloader\tmp\hosts.json
restart jd

could be your host file corrupted? No idea really just wild guess...

I'm going to bed now nite
raztoki

[raztoki]

highly dobutful that the decrypter takes that long when everything goes right. As in time could increase/vary slightly if additional http request caused by connection timeout issues are required, or password prompting or album forwarding takes place. Also if you add links that are not in the following format, imgsrc.ru/username/a\d+.html, more page requests take place in order to make sure it's decrypting on page one of given album. The slow part of the original decrypter was processing of each of the img urls imgsrc.ru/username/(\d+).html (notice no a) to get the direct http url of the image. This is now done in the dedicated hoster plugin. Since my previous change to this plugin (a month or so ago) I placed in a time to decrypt output in the log. Though now displays in seconds and not minutes as the decrypter works fast!

the link you provided in this thread for example
62|imgsrc.ru 29/05/13 12:44:46 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 1 seconds. Returning 43 DownloadLinks for
a few tests of my own
84|imgsrc.ru 29/05/13 1:33:57 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 2 seconds. Returning 68 DownloadLinks for
86|imgsrc.ru 29/05/13 1:34:48 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 1 seconds. Returning 43 DownloadLinks for
114|imgsrc.ru 29/05/13 1:40:30 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 19 seconds. Returning 705 DownloadLinks for
151|imgsrc.ru 29/05/13 1:44:45 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 21 seconds. Returning 898 DownloadLinks for

considering my ping to imgsrc is half a second, performance wise that's a great outcome.

[djmakinera]

That's right, now is the ultra-fast!

Although there is no original names (in some cases the useful since they contain information about the image - if detailed name image*) I know that the name comes from Vkontakte and all the pictures also;)

1. I found a problem with the name (here is not complete title)

example1:
* **External links are only visible to Support Staff****External links are only visible to Support Staff**

missing at the end of the name " .27 "

[raztoki]

Original filenames as in 'володина даша 1.jpg' ?? I didn't wana go down that path as this way at least they will never dupe out with exact same filename. And I moved away from final link name provided by the link itself because its randomised, and they have changed it in the past. They can very quickly change it via dynamic javascript, for instance.
// document.getElementById('oripic').href='**External links are only visible to Support Staff**+o.charAt(0)+'/'+o+'/9/31970729'+n+'.jpg';
// document.getElementById('bigpic').src='**External links are only visible to Support Staff**+o.charAt(0)+'/'+o+'/8/'+'463518'+n+'.jpg';

This way, you can get the saved filename (upid) and go back to its thread if you still know the username eg. imgsrc.ru/username/upid.html.

I traced the vol.27 issue, its actually picked up fine by the plugin. There are cleanup routines somewhere else which remove it. So not a problem of this plugin.

[djmakinera]

I do not know what's the problem. Maybe you need to skip ads or other direction links

Subject category (+18) - there are other links + advertising

**External links are only visible to Support Staff****External links are only visible to Support Staff**


http://jdownloader.org/pastebin/111659


--ID:57TS:1370789906161-13-06-09 16:58:26 - [jd.plugins.decrypter.ImgSrcRu(parsePage)] -> Possible plugin error: Please confirm in your webbrowser that this album

[raztoki]

not a bug buddy, just it couldn't find any additional links its a warning to let you know/confirm in browser!

that album is only one photo, as albums display 48 images, the first image doesn't have the links I am looking for, I warn in case of possible plugin error. If only one pic returned within a one pic album, then theirs nothing to worry about right?