saitejatirunagari Claude Opus 4.8 commited on
Commit
6d1176f
Β·
1 Parent(s): bd9ddce

fix(platforms): visible ever-jobs sidecar status + Node20 ESM CommonJS fix

Browse files

User: 'ran all platforms but only LinkedIn fetched.' Root cause: the 160+
ever-jobs platforms need the NestJS sidecar on :3001, which fails to start
on HF β€” and the failure was SILENT (returned 0, so only direct scrapers
showed results).

Two fixes:
1. Visibility (guaranteed): before adding the ever-jobs scraper, health-check
the sidecar. If UP, log 'sidecar UP β€” N platforms enabled'. If DOWN, log a
loud warning that those N platforms are SKIPPED this run and only direct
scrapers ran. No more silent zero.

2. Node-20 ESM startup attempt: the prior NODE_OPTIONS
'--experimental-specifier-resolution=node' was REMOVED in Node 20, so it
never applied β€” that's why the sidecar kept failing with
ERR_MODULE_NOT_FOUND on extensionless imports. New approach: force ts-node
to emit CommonJS (TS_NODE_COMPILER_OPTIONS module=commonjs +
TS_NODE_TRANSPILE_ONLY), where extensionless relative imports resolve.

Honest note: fix #2 is a best-effort attempt on a third-party NestJS monorepo
I can't run live here; if it still fails, fix #1 ensures the user clearly
sees that the 160+ platforms were skipped (vs silent confusion), and the 6
direct scrapers continue working.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Files changed (2) hide show
  1. start.sh +9 -3
  2. ui.py +12 -1
start.sh CHANGED
@@ -1,8 +1,12 @@
1
  #!/bin/bash
2
  # Start ever-jobs NestJS API in background, then Streamlit.
3
  #
4
- # ever-jobs uses extensionless ESM imports which Node 20 strict mode rejects
5
- # (ERR_MODULE_NOT_FOUND for .../plugin.module). NODE_OPTIONS allows them.
 
 
 
 
6
  #
7
  # If ever-jobs fails anyway, Streamlit still launches and dedicated scrapers
8
  # (LinkedIn/Indeed/Glassdoor/Remotive/WWR/Naukri) keep working β€” only the
@@ -10,7 +14,9 @@
10
 
11
  echo "=== Starting ever-jobs API on port 3001 ==="
12
  cd /app/vendor/ever-jobs
13
- NODE_OPTIONS="--experimental-specifier-resolution=node" npx nest start api > /tmp/ever-jobs.log 2>&1 &
 
 
14
  EVER_JOBS_PID=$!
15
 
16
  # Shorter wait β€” 45s max. ts-node compile on first start usually takes ~20s.
 
1
  #!/bin/bash
2
  # Start ever-jobs NestJS API in background, then Streamlit.
3
  #
4
+ # ever-jobs uses extensionless relative imports (.../plugin.module). Under
5
+ # Node 20, ESM strict mode rejects these with ERR_MODULE_NOT_FOUND, and the
6
+ # old `--experimental-specifier-resolution=node` flag was REMOVED in Node 20.
7
+ # Fix: force ts-node to emit CommonJS, where extensionless imports resolve
8
+ # fine. We also disable type-checking (transpile-only) for faster, more
9
+ # tolerant startup.
10
  #
11
  # If ever-jobs fails anyway, Streamlit still launches and dedicated scrapers
12
  # (LinkedIn/Indeed/Glassdoor/Remotive/WWR/Naukri) keep working β€” only the
 
14
 
15
  echo "=== Starting ever-jobs API on port 3001 ==="
16
  cd /app/vendor/ever-jobs
17
+ export TS_NODE_TRANSPILE_ONLY=1
18
+ export TS_NODE_COMPILER_OPTIONS='{"module":"commonjs","moduleResolution":"node","esModuleInterop":true,"allowJs":true}'
19
+ npx nest start api > /tmp/ever-jobs.log 2>&1 &
20
  EVER_JOBS_PID=$!
21
 
22
  # Shorter wait β€” 45s max. ts-node compile on first start usually takes ~20s.
ui.py CHANGED
@@ -1720,8 +1720,19 @@ if show_config and start and not st.session_state.running:
1720
 
1721
  _ej_platforms = [p for p in _all_plats if p not in _legacy_keys]
1722
  if _ej_platforms:
 
 
 
1723
  from src.scrapers.ever_jobs import EverJobsScraper
1724
- scraper_map["ever_jobs"] = ("EverJobs", EverJobsScraper(_ej_platforms))
 
 
 
 
 
 
 
 
1725
  else:
1726
  _step_skip("ever_jobs")
1727
 
 
1720
 
1721
  _ej_platforms = [p for p in _all_plats if p not in _legacy_keys]
1722
  if _ej_platforms:
1723
+ # The 160+ "ever-jobs" platforms need the NestJS sidecar on
1724
+ # :3001. If it isn't up, be LOUD about it (it silently
1725
+ # returned 0 before, so users saw only LinkedIn results).
1726
  from src.scrapers.ever_jobs import EverJobsScraper
1727
+ from src.ever_jobs_bridge.server import is_running as _ej_health
1728
+ if _ej_health():
1729
+ scraper_map["ever_jobs"] = ("EverJobs", EverJobsScraper(_ej_platforms))
1730
+ _q_log(f"🌐 ever-jobs sidecar UP β€” {len(_ej_platforms)} extra platforms enabled")
1731
+ else:
1732
+ _q_log(f"⚠ ever-jobs sidecar DOWN β€” {len(_ej_platforms)} platforms "
1733
+ f"(Indeed/Glassdoor/Greenhouse/Lever/etc.) SKIPPED this run. "
1734
+ f"Direct scrapers (LinkedIn/Indeed/Glassdoor/Remotive/WWR/Naukri) still run.")
1735
+ _step_skip("ever_jobs")
1736
  else:
1737
  _step_skip("ever_jobs")
1738