JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 16.04.2019, 04:14
emsef emsef is offline
JD Fan
 
Join Date: Jun 2009
Location: UK
Posts: 76
Default How to prevent duplicates in LinkGrabber?

I have a long list of links in some JSON text.

Code:
{"url": "**External links are only visible to Support Staff**, "description": "BOUNCIN 1.0 - ShareWare "}
{"url": "**External links are only visible to Support Staff**, "description": "BOUNCIN 1.0 - ShareWare This game of skill combines elements of Breakout pinball and air hockey. Use your stylus or mouse to move the paddle around the bottom portion of the screen. Use smooth strokes with the paddle to hit the ball up ramps and into blocks and targets. Watch for on-screen messages for hints on how to complete each level. "}
Some of the links are duplicate, even if their surrounding text is different.

When I paste this text into JD, both links are added to the LinkGrabber.
They get a folder of their own.

But I'd like just one link to be grabbed.

How can I prevent duplicates being added at the LinkGrabber stage?

Or how can I clean up (remove) the duplicates?

Thanks!
Reply With Quote
  #2  
Old 16.04.2019, 10:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

JDownloader can't know that those two urls are duplicates because they are different. You have different workarounds:
1.) setup a filter in Settings-Linkgrabber filter and avoid adding the 2nd version of the url, for example sourceURL contains /pilot/ to avoid those urls
2.) add both urls and merge both into the same filepackage and on actual download JDownloader should detect the mirror
3.) make use of Linkcrawler rules to auto-modify/rewrite, use board search for this.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 16.04.2019, 11:09
emsef emsef is offline
JD Fan
 
Join Date: Jun 2009
Location: UK
Posts: 76
Default

OK thanks!

I understand the situation.
Reply With Quote
  #4  
Old 16.04.2019, 13:14
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

You're welcome
In case you've got further questions or need help, just ask again
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 00:03.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.