JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #81  
Old 09.05.2013, 23:51
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default

Interesting :
JD 2 - 30000 photos = 20 hours (if finish parsing error will crash or not)

Tested:
another application - 30000 photos = 1 hours 30 min
Reply With Quote
  #82  
Old 10.05.2013, 09:28
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

on your Original link the album is only 269 images, Took me 2.5minutes on my new code. The only way to make it faster again is to remove half decrypter code and make it into a hoster plugin. Probably the reason why the other program is so much faster, as the adding of links is only small component of the task. If online checking and dl components done its own plugin, even at the download stage you could increase the efficiency of the plugin by about 10 fold, siding scale depending on the size of the gallery. The decrypter does everything one step at a time, and this can effect over all time lapsed assuming large tasks.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #83  
Old 10.05.2013, 11:05
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default

Raztoki - Is JD - first scans page 1, page2, Page3, Page4 etc. ? and then analyzes and then processes the links?

Last edited by djmakinera; 10.05.2013 at 11:58.
Reply With Quote
  #84  
Old 10.05.2013, 13:14
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

yah, not sure why we don't use the all image page (/main/pic_tape.php?ad=uid), think its always the biggest image, we would just need to goto x pages vs, x * y to get finallinks. I'll ask psp later, maybe they do not show all images/pic tape for all albums?
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #85  
Old 10.05.2013, 14:05
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default

Do not use (/main/pic_tape.php?ad=uid)
because if the image is large in size (full size) - does not show the maximum size

**External links are only visible to Support Staff****External links are only visible to Support Staff** (NORMAL SIZE) (/main/pic_tape.php?ad=uid)

**External links are only visible to Support Staff****External links are only visible to Support Staff** (FULL SIZE)

Last edited by djmakinera; 10.05.2013 at 14:10.
Reply With Quote
  #86  
Old 10.05.2013, 16:37
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

I compared the link returned from the current implementation in JD and then went to pic_tape to find the same image and it was the exact same image url. So not sure what you did but I can't reproduce your findings.

If we use this method it will return all links as you only need to hit each 'x pic tape page' (* 12 a page) = total images vs 'each album page y * (48 -1) each image page = total images', in the standard album layout.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 10.05.2013 at 16:43.
Reply With Quote
  #87  
Old 10.05.2013, 16:42
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default

Example:

_**External links are only visible to Support Staff**Red">b0.eu.IMGSRC (NORMAL)

_**External links are only visible to Support Staff**Red">o0.eu.IMGSRC (FULL)

Last edited by djmakinera; 10.05.2013 at 16:44.
Reply With Quote
  #88  
Old 10.05.2013, 17:17
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

I saw that, and that was not my findings.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #89  
Old 21.05.2013, 20:14
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default iMSRC.RU - broken

**External links are only visible to Support Staff****External links are only visible to Support Staff**

logfile:

http://jdownloader.org/pastebin/110969
Reply With Quote
  #90  
Old 26.05.2013, 18:09
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

Seems it only happens for new links, and I made support based from the html in your log. That said I was able to find some new links on the site also to test it with, and it works. In the process of rewriting the plugin to speed up efficiency. Just need to find some passworded links so I can implement/test the new code!
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #91  
Old 28.05.2013, 22:18
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default

raztoki
Still loses to time hehe

Test:

JD (12 photos)
31 second
47 second

Other program (12 photos)
3 second
4 second
Reply With Quote
  #92  
Old 28.05.2013, 22:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 47,654
Default

JD is no site crawler/ripper, at least it's not made exactly for that.

GreeZ pspzockerscene
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #93  
Old 28.05.2013, 22:34
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default

I know, but it was addressed to raztoki. We have a common theme hehe
Reply With Quote
  #94  
Old 28.05.2013, 23:39
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default "Broken parsing" (probably all links!)

"Broken parsing" (probably all links)

I do not know what happened but JD2 shows that all of the links "out of date"
Example:

Vk.com - broken
Imgsrc.ru - broken
zippyshare.com - broken
etc...



I tested different installations of JD2 (without updating!) - All the links work.
So what happened?


remove hosts and works all

Last edited by djmakinera; 28.05.2013 at 23:59.
Reply With Quote
  #95  
Old 29.05.2013, 07:36
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

not sure why I didn't post here

so answer for everyone else is..

Quote:
Originally Posted by raztoki
try closing jd
goto install folder
delete that file jdownloader\tmp\hosts.json
restart jd

could be your host file corrupted? No idea really just wild guess...

I'm going to bed now nite
raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #96  
Old 29.05.2013, 07:46
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

highly dobutful that the decrypter takes that long when everything goes right. As in time could increase/vary slightly if additional http request caused by connection timeout issues are required, or password prompting or album forwarding takes place. Also if you add links that are not in the following format, imgsrc.ru/username/a\d+.html, more page requests take place in order to make sure it's decrypting on page one of given album. The slow part of the original decrypter was processing of each of the img urls imgsrc.ru/username/(\d+).html (notice no a) to get the direct http url of the image. This is now done in the dedicated hoster plugin. Since my previous change to this plugin (a month or so ago) I placed in a time to decrypt output in the log. Though now displays in seconds and not minutes as the decrypter works fast!

the link you provided in this thread for example
62|imgsrc.ru 29/05/13 12:44:46 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 1 seconds. Returning 43 DownloadLinks for
a few tests of my own
84|imgsrc.ru 29/05/13 1:33:57 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 2 seconds. Returning 68 DownloadLinks for
86|imgsrc.ru 29/05/13 1:34:48 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 1 seconds. Returning 43 DownloadLinks for
114|imgsrc.ru 29/05/13 1:40:30 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 19 seconds. Returning 705 DownloadLinks for
151|imgsrc.ru 29/05/13 1:44:45 PM - INFO [ jd.plugins.decrypter.ImgSrcRu(decryptIt) ] -> Time to decrypt : 21 seconds. Returning 898 DownloadLinks for

considering my ping to imgsrc is half a second, performance wise that's a great outcome.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 29.05.2013 at 09:04.
Reply With Quote
  #97  
Old 29.05.2013, 09:13
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default

That's right, now is the ultra-fast!

Although there is no original names (in some cases the useful since they contain information about the image - if detailed name image*) I know that the name comes from Vkontakte and all the pictures also;)

1. I found a problem with the name (here is not complete title)

example1:
* **External links are only visible to Support Staff****External links are only visible to Support Staff**

missing at the end of the name " .27 "

Last edited by djmakinera; 29.05.2013 at 09:36.
Reply With Quote
  #98  
Old 29.05.2013, 11:26
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

Original filenames as in 'володина даша 1.jpg' ?? I didn't wana go down that path as this way at least they will never dupe out with exact same filename. And I moved away from final link name provided by the link itself because its randomised, and they have changed it in the past. They can very quickly change it via dynamic javascript, for instance.
// document.getElementById('oripic').href='**External links are only visible to Support Staff**+o.charAt(0)+'/'+o+'/9/31970729'+n+'.jpg';
// document.getElementById('bigpic').src='**External links are only visible to Support Staff**+o.charAt(0)+'/'+o+'/8/'+'463518'+n+'.jpg';

This way, you can get the saved filename (upid) and go back to its thread if you still know the username eg. imgsrc.ru/username/upid.html.

I traced the vol.27 issue, its actually picked up fine by the plugin. There are cleanup routines somewhere else which remove it. So not a problem of this plugin.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #99  
Old 09.06.2013, 18:08
djmakinera djmakinera is offline
JD Legend
 
Join Date: May 2010
Location: Poland
Posts: 7,709
Default imgsrc.ru - not working

I do not know what's the problem. Maybe you need to skip ads or other direction links

Subject category (+18) - there are other links + advertising

**External links are only visible to Support Staff****External links are only visible to Support Staff**


http://jdownloader.org/pastebin/111659


--ID:57TS:1370789906161-13-06-09 16:58:26 - [jd.plugins.decrypter.ImgSrcRu(parsePage)] -> Possible plugin error: Please confirm in your webbrowser that this album

Last edited by djmakinera; 09.06.2013 at 18:10.
Reply With Quote
  #100  
Old 09.06.2013, 19:09
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 15,953
Default

not a bug buddy, just it couldn't find any additional links its a warning to let you know/confirm in browser!

that album is only one photo, as albums display 48 images, the first image doesn't have the links I am looking for, I warn in case of possible plugin error. If only one pic returned within a one pic album, then theirs nothing to worry about right?
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 12:08.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.