Spaces:
Sleeping
fix(platforms): visible ever-jobs sidecar status + Node20 ESM CommonJS fix
Browse filesUser: 'ran all platforms but only LinkedIn fetched.' Root cause: the 160+
ever-jobs platforms need the NestJS sidecar on :3001, which fails to start
on HF β and the failure was SILENT (returned 0, so only direct scrapers
showed results).
Two fixes:
1. Visibility (guaranteed): before adding the ever-jobs scraper, health-check
the sidecar. If UP, log 'sidecar UP β N platforms enabled'. If DOWN, log a
loud warning that those N platforms are SKIPPED this run and only direct
scrapers ran. No more silent zero.
2. Node-20 ESM startup attempt: the prior NODE_OPTIONS
'--experimental-specifier-resolution=node' was REMOVED in Node 20, so it
never applied β that's why the sidecar kept failing with
ERR_MODULE_NOT_FOUND on extensionless imports. New approach: force ts-node
to emit CommonJS (TS_NODE_COMPILER_OPTIONS module=commonjs +
TS_NODE_TRANSPILE_ONLY), where extensionless relative imports resolve.
Honest note: fix #2 is a best-effort attempt on a third-party NestJS monorepo
I can't run live here; if it still fails, fix #1 ensures the user clearly
sees that the 160+ platforms were skipped (vs silent confusion), and the 6
direct scrapers continue working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@@ -1,8 +1,12 @@
|
|
| 1 |
#!/bin/bash
|
| 2 |
# Start ever-jobs NestJS API in background, then Streamlit.
|
| 3 |
#
|
| 4 |
-
# ever-jobs uses extensionless
|
| 5 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
#
|
| 7 |
# If ever-jobs fails anyway, Streamlit still launches and dedicated scrapers
|
| 8 |
# (LinkedIn/Indeed/Glassdoor/Remotive/WWR/Naukri) keep working β only the
|
|
@@ -10,7 +14,9 @@
|
|
| 10 |
|
| 11 |
echo "=== Starting ever-jobs API on port 3001 ==="
|
| 12 |
cd /app/vendor/ever-jobs
|
| 13 |
-
|
|
|
|
|
|
|
| 14 |
EVER_JOBS_PID=$!
|
| 15 |
|
| 16 |
# Shorter wait β 45s max. ts-node compile on first start usually takes ~20s.
|
|
|
|
| 1 |
#!/bin/bash
|
| 2 |
# Start ever-jobs NestJS API in background, then Streamlit.
|
| 3 |
#
|
| 4 |
+
# ever-jobs uses extensionless relative imports (.../plugin.module). Under
|
| 5 |
+
# Node 20, ESM strict mode rejects these with ERR_MODULE_NOT_FOUND, and the
|
| 6 |
+
# old `--experimental-specifier-resolution=node` flag was REMOVED in Node 20.
|
| 7 |
+
# Fix: force ts-node to emit CommonJS, where extensionless imports resolve
|
| 8 |
+
# fine. We also disable type-checking (transpile-only) for faster, more
|
| 9 |
+
# tolerant startup.
|
| 10 |
#
|
| 11 |
# If ever-jobs fails anyway, Streamlit still launches and dedicated scrapers
|
| 12 |
# (LinkedIn/Indeed/Glassdoor/Remotive/WWR/Naukri) keep working β only the
|
|
|
|
| 14 |
|
| 15 |
echo "=== Starting ever-jobs API on port 3001 ==="
|
| 16 |
cd /app/vendor/ever-jobs
|
| 17 |
+
export TS_NODE_TRANSPILE_ONLY=1
|
| 18 |
+
export TS_NODE_COMPILER_OPTIONS='{"module":"commonjs","moduleResolution":"node","esModuleInterop":true,"allowJs":true}'
|
| 19 |
+
npx nest start api > /tmp/ever-jobs.log 2>&1 &
|
| 20 |
EVER_JOBS_PID=$!
|
| 21 |
|
| 22 |
# Shorter wait β 45s max. ts-node compile on first start usually takes ~20s.
|
|
@@ -1720,8 +1720,19 @@ if show_config and start and not st.session_state.running:
|
|
| 1720 |
|
| 1721 |
_ej_platforms = [p for p in _all_plats if p not in _legacy_keys]
|
| 1722 |
if _ej_platforms:
|
|
|
|
|
|
|
|
|
|
| 1723 |
from src.scrapers.ever_jobs import EverJobsScraper
|
| 1724 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1725 |
else:
|
| 1726 |
_step_skip("ever_jobs")
|
| 1727 |
|
|
|
|
| 1720 |
|
| 1721 |
_ej_platforms = [p for p in _all_plats if p not in _legacy_keys]
|
| 1722 |
if _ej_platforms:
|
| 1723 |
+
# The 160+ "ever-jobs" platforms need the NestJS sidecar on
|
| 1724 |
+
# :3001. If it isn't up, be LOUD about it (it silently
|
| 1725 |
+
# returned 0 before, so users saw only LinkedIn results).
|
| 1726 |
from src.scrapers.ever_jobs import EverJobsScraper
|
| 1727 |
+
from src.ever_jobs_bridge.server import is_running as _ej_health
|
| 1728 |
+
if _ej_health():
|
| 1729 |
+
scraper_map["ever_jobs"] = ("EverJobs", EverJobsScraper(_ej_platforms))
|
| 1730 |
+
_q_log(f"π ever-jobs sidecar UP β {len(_ej_platforms)} extra platforms enabled")
|
| 1731 |
+
else:
|
| 1732 |
+
_q_log(f"β ever-jobs sidecar DOWN β {len(_ej_platforms)} platforms "
|
| 1733 |
+
f"(Indeed/Glassdoor/Greenhouse/Lever/etc.) SKIPPED this run. "
|
| 1734 |
+
f"Direct scrapers (LinkedIn/Indeed/Glassdoor/Remotive/WWR/Naukri) still run.")
|
| 1735 |
+
_step_skip("ever_jobs")
|
| 1736 |
else:
|
| 1737 |
_step_skip("ever_jobs")
|
| 1738 |
|