[MediaFanatic]

I've spent 4-5 hours looking through posts, googling, and testing dozens of different LinkCrawler variants, but it has become clear I have no idea what I'm doing, lol

The Situation:
I have a valid, paid LinkedIn account. I would like to archive some courses offline, by downloading with jDownloader (build date 4/4/2020).

LinkedIn requires a valid login (not HTTP auth) to access the embedded .MP4 videos. My goal is to pass my login credentials using the LinkCrawler rules, where I paste the exact session variables being used to access the videos from Chrome browser.

Two example links:
(obfuscated to allow posting)

w-w-w.linkedin.c-o-m/learning/autocad-architecture-2015-essential-training/setting-door-and-window-styles?autoplay=true
w-w-w.linkedin.c-o-m/learning/autocad-architecture-2015-essential-training/working-with-columns?autoplay=true

When loggged-in (valid session cookie), these links have an embedded *.mp4 video from a lynda.com url.

I am passing the cookies from my valid login using the LinkCrawler Rules at:
Settings > Advanced Settings > LinkCrawler: Linkcrawler Rules

I also pass my user-account/password. I do not believe this is used, because LinkedIn does not use HTTP auth.

I am pasting my LinkCrawler code, below
(where you see "removed", I have removed the data, either for privacy or to shorten / easier to read):

[ {
  "id" : 1586007825050,
  "name" : "LinkedInVideos",
  "enabled" : true,
  "logging" : true,
  "maxDecryptDepth" : 1,
  "domainPattern" : ".*linkedin\\.com",
  "filenamePattern" : "\\.mp4$",
  "rule" : "DEEPDECRYPT",
  "accountPattern" : "removed@removed.com",
  "passwordPattern" : "removed",
  "packageNamePattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "cookies" : [ [ "215E3D5995C57C0A495C55%40AdobeOrg", "removed" ], [ "AMCV_14215E3D5995C57C0A495C55%40AdobeOrg", "removed" ], [ "JSESSIONID", "removed" ], [ "PLAY_LANG", "en" ], [ "PLAY_SESSION", "removed" ], [ "SID", "removed" ], [ "VID", "removed" ], [ "_ga", "removed" ], [ "_gcl_au", "removed" ], [ "aam_uuid", "removed" ], [ "bcookie", "removed" ], [ "bscookie", "removed" ], [ "lang", "v=2&lang=en-US" ], [ "li_at", "removed" ], [ "liap", "TRUE" ], [ "lidc", "removed" ], [ "lil-lang", "en_US" ], [ "lissc", "1" ], [ "lissc1", "1" ], [ "lissc2", "1" ], [ "sl", "v=1&85fMp" ] ],
  "updateCookies" : true
} ]

I chose the DEEPDECRYPT method (with a depth of 1). The reason for this, the link I am pasting may embed or redirect to Lynda. To be safe, I felt it should do a "deep" review of the site, but I could be wrong. I also tested the DIRECTHTTP method, but that didn't work, either.

I've spent all day trying different things and I'm out of ideas. Are there any LinkCrawler experts out there who can help?

Thank you in advance!

[raztoki]

how is the content provided iframe or links to direct source?

deepPattern should be the embedded link to lynda.com or the mp4 or what ever is available.

Pattern should be used for linkedin to match the url format you need to support, try not to create a pattern for entire site, else it will be triggered for no reason.

cookies should work, and only used for the crawling. What do you see in the crawler log?

raztoki

[MediaFanatic]

@raztoki, thaks so much for the reply.

I'm including my replies to your comments, in-line:

how is the content provided iframe or links to direct source?

It is embedded within the actual URL's I'm providing to jDownloader. I'm including an HTML snippet of the video-container, if you're interested. For readability, I have removed the long GUID and extraneous parameters:

<div data-vjs-player="" aria-label="Video player">
	<video class="vjs-tech" id="vjs_video_3_html5_api" tabindex="-1" role="application" preload="auto" src="**External links are only visible to Support Staff**></video>
</div>

deepPattern should be the embedded link to lynda.com or the mp4 or what ever is available.

I think you may be suggesting that I replace the "filePattern" with the "deepPattern"?

For example, should I remove this:
"filenamePattern" : "\\.mp4$",
and replace with this:
"deepPattern" : "\\.mp4$",

Pattern should be used for linkedin to match the url format you need to support, try not to create a pattern for entire site, else it will be triggered for no reason.

I had thought I was doing as you say, by using:
"domainPattern" : ".*linkedin\\.com",

Is this wrong / should I be using a different parameter?

What do you see in the crawler log?

The log shows that LinkedIn/Lynda are not receiving the cookies (or they are not being honored). This is not explicitly show. Rather, it's based on the log showing the redirects to non-authenticated page, without access to the videos.

If I run the same test on a "free" video, it works fine, further evidence that the cookies are not being passed properly.

I'm including all entries in the LinkCrawler log, related to one single call. You'll see many requests -- this is a result of the non-authenticated redirects, IMHO.

------------------------Thread: 121:LinkCrawler.log-----------------------
--ID:121TS:1586067925892-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&session_redirect=|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925893-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-join|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925894-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&session_redirect=https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-join|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925894-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&session_redirect=|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925894-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-join|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925895-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&session_redirect=https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-join|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925895-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&session_redirect=|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925895-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-join|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925898-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&fromSignIn=true&session_redirect=|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925898-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-signin|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925898-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&fromSignIn=true&session_redirect=https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-signin|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925899-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&fromSignIn=true&session_redirect=|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925899-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-signin|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925899-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&fromSignIn=true&session_redirect=https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-signin|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925899-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&fromSignIn=true&session_redirect=|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925899-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training&trk=learning-course_nav-header-signin|https://www.linkedin.com/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925914-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_course_info_buy_course_upsell_subscription&trk=lil_upsell_course_info_buy_course_upsell_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925915-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_course_info_buy_course_upsell_subscription&trk=lil_upsell_course_info_buy_course_upsell_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925916-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_course_info_buy_course_upsell_subscription&trk=lil_upsell_course_info_buy_course_upsell_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067925968-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&source=subs_learn_start&trk=sign-in-modal_learning-sign-in-card|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925969-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&source=subs_learn_start&trk=sign-in-modal_learning-sign-in-card|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067925969-4/5/20 2:25:25 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training&source=subs_learn_start&trk=sign-in-modal_learning-sign-in-card|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training
--ID:121TS:1586067926272-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_exercise_file_subscription&trk=lil_upsell_exercise_file_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926273-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_exercise_file_subscription&trk=lil_upsell_exercise_file_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926274-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_exercise_file_subscription&trk=lil_upsell_exercise_file_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926277-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_exercise_file_subscription&trk=lil_upsell_exercise_file_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926278-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_exercise_file_subscription&trk=lil_upsell_exercise_file_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926279-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_exercise_file_subscription&trk=lil_upsell_exercise_file_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926293-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_offline_viewing_subscription&trk=lil_upsell_offline_viewing_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926294-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows&lipi=urn:li:page:d_d_learning_course_video_guest%3BVGxNbf8vQNSI90H3pDwP/Q==&upsellTrk=lil_upsell_offline_viewing_subscription&trk=lil_upsell_offline_viewing_subscription|https://www%2Elinkedin%2Ecom/learning/autocad-architecture-2015-essential-training/adding-doors-and-windows
--ID:121TS:1586067926295-4/5/20 2:25:26 AM -  [jd.parser.html.HTMLParser(logInfo)] -> Apply auto cut off|**External links are only visible to Support Staff**

Looking at my parameter-string, would it be possible for you to tell me, if you see anything specific that I should change?

Or is there any way for me to dig deeper t confirm cookies are being passed properly, etc? Hopefully something a bit quicker than installing Wireshark?

Thank you again for your help!

[raztoki]

pattern say "https?://(?:\w+\.)?linkedin\.com/learning/.+"
deepPattern say "https?://[^/]+/.*?\.mp4.*?"
domainPattern isn't a thing far as I'm aware re: https://board.jdownloader.org/showth...280#post422008
filename pattern is used for a desired filename that isn't the name within the url. So you find something within html.
lynda wont need those those cookies?? right they are from linkedin, since its bound to the domain (linkedin.com). The hash (guid) in the url which what should give you access

[MediaFanatic]

I forgot to provide you with important information that causes me to believe that cookies are not working as I expect in LinkCrawler Rules....

1. I've tested the same cookies (formatted in a Netscape cookie.txt file) with youtube-dl and it works perfectly.

2. As mentioned above, everything works fine with jDownloader, when downloading a video that does not require login.

3. The LinkCrawler log shows that we're being re-directed because we're not authenticated (we haven't submitted a valid session).

This leads me to a question...

Can you think of any reason why Youtube-DL will work with the same cookies, but jDownloader does not?

Thank you.

[MediaFanatic]

Sorry, we posted at the same time.

A couple quick replies:

(1) I believe the issue is 100% related to cookies. Keep in mind, my pattern does work when the authentication/cookies are not required.

(2) I cannot see your "pattern" that you provided, because the this forum replaced the text, blocking URL's.

(3) In this case, the .mp4 is not in the filename, it's in the embedded content, so I'll use "filenamePattern" for the .MP4 (instead of deepPattern), is that correct?

Thank you again.

[raztoki]

not sure about the cookie session differences between youtube-dl(never used it) and JD.
I personally think cookie implementation in crawler rules isn't thought out well, there surely needs to be domain reference mandatory. I mean what if your pattern is multidomain, yet cookie is applied to every request within that rule. I don't like Cookie implementation within JD Browser, but thats another story.

Sorry about the url not showing for linkedin see if that works (figured a way to bypass it)