fix: HF Space proxy startup — CORS flags, port conflict, playwright deps

Three bugs prevented the FastAPI+Streamlit reverse proxy from working on HF:

1. Remove STREAMLIT_SERVER_PORT=7860 + STREAMLIT_SERVER_ADDRESS=0.0.0.0 from
Dockerfile ENV — these were left over from the old single-process architecture
and conflict with the proxy design (uvicorn on 7860, Streamlit on 8501).

2. Add --server.enableCORS false --server.enableXsrfProtection false to the
Streamlit subprocess in start_streamlit() — Streamlit's default CORS middleware
rejects requests whose Origin doesn't match 127.0.0.1:8501, which breaks every
request forwarded by the FastAPI proxy from the public HF Space domain.

3. Remove --with-deps from _ensure_playwright() in ui.py (timeout 120s→30s) —
apt-get requires root; failing silently as uid 1000 on HF added wasted startup
latency on every cold boot. Chromium is pre-installed in the Dockerfile.

4. Add company_ats to PIPELINE_STEPS so it appears in the run progress UI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Dockerfile

@@ -85,10 +85,11 @@ RUN chmod +x start.sh
 # Expose Streamlit port
 EXPOSE 7860
 
-ENV STREAMLIT_SERVER_PORT=7860 \
-    STREAMLIT_SERVER_ADDRESS=0.0.0.0 \
-    STREAMLIT_SERVER_HEADLESS=true \
+# Only keep non-conflicting Streamlit env vars.
+# STREAMLIT_SERVER_PORT and STREAMLIT_SERVER_ADDRESS are intentionally omitted:
+# they would conflict with the proxy architecture where api_server.py runs
+# uvicorn on 7860 and Streamlit is started on 127.0.0.1:8501 via CLI args.
+ENV STREAMLIT_SERVER_HEADLESS=true \
     STREAMLIT_BROWSER_GATHER_USAGE_STATS=false
 
-# start.sh boots ever-jobs on port 3001, then launches Streamlit on 7860
 CMD ["./start.sh"]

HISTORY.md

@@ -4,6 +4,36 @@ A running log of everything built, fixed, and changed. Most recent first.
 
 ---
 
+## 2026-06-23 (PM5) — Fix HF Space startup: CORS proxy flags + port conflict + playwright
+
+Three bugs prevented the FastAPI + Streamlit proxy from working on HF Spaces:
+
+1. **`STREAMLIT_SERVER_PORT=7860` removed from Dockerfile ENV** — this env var
+   conflicted with the proxy architecture: `api_server.py` runs uvicorn on 7860
+   and Streamlit on 8501 (via CLI `--server.port 8501`). If Streamlit used the
+   env var instead of the CLI arg, it would try to bind to 7860 while uvicorn
+   already held that port, causing one of them to crash.
+
+2. **`--server.enableCORS false --server.enableXsrfProtection false` added to
+   `start_streamlit()`** — Streamlit's default CORS middleware rejects requests
+   whose `Origin` header doesn't match `127.0.0.1:8501`. The FastAPI proxy
+   forwards the browser's `Origin` (`https://*.hf.space`) to Streamlit, which
+   then rejects it. Disabling these checks is the standard practice for running
+   Streamlit behind a reverse proxy.
+
+3. **`_ensure_playwright()` in `ui.py`: removed `--with-deps` + reduced timeout
+   from 120s → 30s** — `playwright install chromium --with-deps` runs
+   `apt-get install` for system deps, which requires root. Running as `user`
+   (uid 1000 on HF) it fails silently, but the command would attempt downloads
+   before failing, adding pointless latency on every cold start. Browser is
+   pre-installed in the Dockerfile so `--with-deps` is never needed on HF.
+
+4. **`company_ats` added to `PIPELINE_STEPS`** — added in the Scrapling commit
+   as a scraper but missing from the step-tracker list, so it never showed up
+   in the run progress UI.
+
+---
+
 ## 2026-06-23 (PM4) — ever-jobs sidecar disabled by default (clean/fast startup)
 
 User's HF log showed the ever-jobs Node sidecar crashing on boot

api_server.py

@@ -16,12 +16,14 @@ import base64
 import hashlib
 import hmac
 import io
+import json
 import os
 import re
 import shutil
 import subprocess
 import sys
 import tempfile
+import time
 
 import httpx
 import uvicorn
@@ -747,6 +749,25 @@ async def repair_with_feedback(
 # ── Streamlit subprocess ──────────────────────────────────────────────────────
 _streamlit_proc: subprocess.Popen | None = None
 _STREAMLIT_PORT = 8501
+_DEBUG_LOG_PATH = "debug-03623a.log"
+
+
+def _debug_log(run_id: str, hypothesis_id: str, location: str, message: str, data: dict) -> None:
+    """Append one NDJSON runtime debug event for this session."""
+    payload = {
+        "sessionId": "03623a",
+        "runId": run_id,
+        "hypothesisId": hypothesis_id,
+        "location": location,
+        "message": message,
+        "data": data,
+        "timestamp": int(time.time() * 1000),
+    }
+    try:
+        with open(_DEBUG_LOG_PATH, "a", encoding="utf-8") as f:
+            f.write(json.dumps(payload, separators=(",", ":")) + "\n")
+    except Exception:
+        pass
 
 
 def start_streamlit() -> None:
@@ -765,8 +786,24 @@ def start_streamlit() -> None:
             "127.0.0.1",
             "--server.headless",
             "true",
+            # Required for reverse-proxy operation: without these, Streamlit's
+            # CORS/XSRF middleware rejects requests whose Origin header comes
+            # from the public HF Space domain rather than 127.0.0.1:8501.
+            "--server.enableCORS",
+            "false",
+            "--server.enableXsrfProtection",
+            "false",
         ]
     )
+    # #region agent log
+    _debug_log(
+        run_id="open-check-1",
+        hypothesis_id="H4",
+        location="api_server.py:start_streamlit",
+        message="streamlit_process_started",
+        data={"pid": _streamlit_proc.pid, "port": _STREAMLIT_PORT},
+    )
+    # #endregion
 
 
 # ── WebSocket proxy for /_stcore/stream (Streamlit live updates) ──────────────
@@ -780,6 +817,15 @@ async def ws_proxy(websocket: WebSocket):
     import websockets.exceptions as _ws_exc
 
     await websocket.accept()
+    # #region agent log
+    _debug_log(
+        run_id="open-check-1",
+        hypothesis_id="H3",
+        location="api_server.py:ws_proxy",
+        message="ws_client_connected",
+        data={"client": str(websocket.client)},
+    )
+    # #endregion
     upstream_url = (
         f"ws://127.0.0.1:{_STREAMLIT_PORT}/_stcore/stream"
         f"?{websocket.scope.get('query_string', b'').decode()}"
@@ -815,8 +861,16 @@ async def ws_proxy(websocket: WebSocket):
 
             await asyncio.gather(client_to_upstream(), upstream_to_client())
 
-    except Exception:
-        pass
+    except Exception as exc:
+        # #region agent log
+        _debug_log(
+            run_id="open-check-1",
+            hypothesis_id="H3",
+            location="api_server.py:ws_proxy",
+            message="ws_upstream_connect_or_proxy_failed",
+            data={"error": str(exc)[:200]},
+        )
+        # #endregion
     finally:
         try:
             await websocket.close()
@@ -837,6 +891,15 @@ async def proxy(request: Request, path: str):
     if path == "api" or path.startswith("api/"):
         return JSONResponse({"error": "not_found", "path": "/" + path}, status_code=404)
 
+    # #region agent log
+    _debug_log(
+        run_id="open-check-1",
+        hypothesis_id="H2",
+        location="api_server.py:proxy_entry",
+        message="proxy_request_received",
+        data={"method": request.method, "path": path},
+    )
+    # #endregion
     url = f"http://127.0.0.1:{_STREAMLIT_PORT}/{path}"
     params = dict(request.query_params)
     headers = {
@@ -860,13 +923,35 @@ async def proxy(request: Request, path: str):
                      "connection", "keep-alive"}
             clean_headers = {k: v for k, v in resp.headers.items()
                              if k.lower() not in _DROP}
+            # #region agent log
+            _debug_log(
+                run_id="open-check-1",
+                hypothesis_id="H2",
+                location="api_server.py:proxy_response",
+                message="proxy_upstream_response",
+                data={
+                    "path": path,
+                    "status_code": resp.status_code,
+                    "content_type": resp.headers.get("content-type", ""),
+                },
+            )
+            # #endregion
             return StarletteResponse(
                 content=resp.content,
                 status_code=resp.status_code,
                 headers=clean_headers,
                 media_type=resp.headers.get("content-type"),
             )
-        except httpx.ConnectError:
+        except httpx.ConnectError as exc:
+            # #region agent log
+            _debug_log(
+                run_id="open-check-1",
+                hypothesis_id="H4",
+                location="api_server.py:proxy_connect_error",
+                message="streamlit_connect_error",
+                data={"path": path, "error": str(exc)[:200]},
+            )
+            # #endregion
             # Streamlit may still be starting up — return a friendly retry message.
             return StarletteResponse(
                 content=b"Streamlit is starting, please wait...",

ui.py

@@ -616,12 +616,15 @@ hr { border-color: #E2E8F0 !important; opacity: 0.5 !important; }
 """, unsafe_allow_html=True)
 
 # ── Playwright install (runs once per server lifetime on HF Spaces) ──────────
+# On HF Spaces, Chromium is pre-installed in the Dockerfile so this is a fast
+# no-op check. We omit --with-deps because system-package installs require root
+# and would fail silently, adding ~10s of pointless startup latency every boot.
 @st.cache_resource(show_spinner=False)
 def _ensure_playwright():
     import subprocess, sys as _sys
     result = subprocess.run(
-        [_sys.executable, "-m", "playwright", "install", "chromium", "--with-deps"],
-        capture_output=True, text=True, timeout=120,
+        [_sys.executable, "-m", "playwright", "install", "chromium"],
+        capture_output=True, text=True, timeout=30,
     )
     return result.returncode == 0
 
@@ -669,6 +672,7 @@ PIPELINE_STEPS = [
     {"id": "remotive",        "icon": "🌍", "title": "Remotive"},
     {"id": "weworkremotely",  "icon": "💻", "title": "WeWorkRemotely"},
     {"id": "naukri",          "icon": "🇮🇳", "title": "Naukri"},
+    {"id": "company_ats",     "icon": "🏢", "title": "Company ATS"},
     {"id": "ever_jobs",       "icon": "🌐", "title": "EverJobs (160+)"},
     {"id": "assess",          "icon": "🤖", "title": "AI Assessment"},
     {"id": "resumes",         "icon": "📝", "title": "Generate Resumes"},