# SkillProbe

SkillProbe tests whether an agent skill actually works: give it any SKILL.md by URL or as text, and it returns a machine-readable conformance report — documentation checks, live probes of every documented endpoint, a 0–100 score, and a concrete fix hint for every failure.

Base URL (use exactly this, with no trailing slash):

https://practical-fulfillment-production-9e7f.up.railway.app

## When to use SkillProbe

- Before depending on a third-party skill: verify its SKILL.md is complete and its endpoints are actually live.
- After you (an agent) write or deploy a service: validate your own SKILL.md and fix every failed check before publishing it.
- When a skill call fails mysteriously: probe the base URL to distinguish "service is down" from "I called it wrong".

## When NOT to use SkillProbe

- Do not use it to fetch arbitrary web pages; it only probes services described by a SKILL.md, plus single-URL reachability checks.
- Do not use it for services on private networks; private, loopback, and internal addresses are rejected by design.
- Do not use it to load-test a service; SkillProbe sends at most 10 requests per validation.

## Authentication

None. All endpoints are public. Do not send credentials.

## Endpoints

### 1. GET /health

Purpose: confirm SkillProbe itself is alive. Call this first if any other call fails.

Parameters: none.

Request:

```bash
curl -s https://practical-fulfillment-production-9e7f.up.railway.app/health
```

Response (HTTP 200, application/json):

```json
{"status": "ok", "service": "skillprobe", "version": "1.0.0", "time": 1752192000}
```

Errors: none expected. A network failure means the service is temporarily down; wait 30 seconds and retry once.

### 2. POST /v1/validate

Purpose: validate a SKILL.md and probe the live service it describes. This is the main endpoint.

Content-Type: application/json.

Body fields (send exactly one of `skill_url` or `skill_markdown`):

| Field | Type | Required | Constraints | Meaning |
|---|---|---|---|---|
| `skill_url` | string | one of the two | http(s), public host, ≤ 2000 chars | URL returning the raw SKILL.md Markdown |
| `skill_markdown` | string | one of the two | ≤ 200000 chars | The SKILL.md content itself |
| `execute` | boolean | no (default `true`) | — | `true` = also call the documented endpoints; `false` = documentation checks only |

Request (validate a hosted SKILL.md):

```bash
curl -s -X POST https://practical-fulfillment-production-9e7f.up.railway.app/v1/validate \
  -H "Content-Type: application/json" \
  -d '{"skill_url": "https://practical-fulfillment-production-9e7f.up.railway.app/skill.md", "execute": true}'
```

Request (validate inline Markdown, static checks only):

```bash
curl -s -X POST https://practical-fulfillment-production-9e7f.up.railway.app/v1/validate \
  -H "Content-Type: application/json" \
  -d '{"skill_markdown": "# My Service\nDoes things.\nhttps://myservice.example\n\nGET /health", "execute": false}'
```

Response (HTTP 200, application/json; fields are stable):

```json
{
  "report_id": "rep_1a2b3c4d5e6f7a8b",
  "cached": false,
  "mode": "static+live",
  "source": {"type": "url", "url": "https://example-skill.app/skill.md", "final_url": "https://example-skill.app/skill.md"},
  "score": 92,
  "grade": "A",
  "passed": true,
  "summary": "12/13 checks passed; score 92/100 (A).",
  "skill": {"title": "My Service", "base_url": "https://example-skill.app", "endpoints_documented": 3, "curl_examples": 2},
  "checks": [
    {"id": "static.title", "severity": "error", "passed": true, "detail": "title: 'My Service'", "hint": "Start the file with a single '# Service Name' heading."}
  ],
  "endpoints": [
    {"method": "GET", "path": "/health", "url": "https://example-skill.app/health", "probed": true, "status_code": 200, "latency_ms": 143, "content_type": "application/json", "is_json": true, "error": null, "skipped_reason": null, "matched_example_keys": ["status"], "missing_example_keys": []}
  ],
  "duration_ms": 1912,
  "limits": {"max_live_requests": 10, "max_markdown_bytes": 200000}
}
```

How to read the report:

- `passed` is `true` when the score is at least 75 and no error-severity check failed. Treat `passed: true` as "safe to depend on this skill".
- Every object in `checks` has `passed`, `severity` (`error` > `warning` > `info`), `detail`, and a `hint` that states the exact fix.
- Every object in `endpoints` shows what was actually called and what came back. `skipped_reason` is non-null when an endpoint was not called (for example, a write endpoint with no documented example body).
- `report_id` is deterministic for identical input: repeating the same request returns the cached report with `"cached": true`. This makes retries safe.

Errors (all errors use this exact object shape):

```json
{"error": {"code": "skill_fetch_failed", "message": "could not fetch skill_url: timeout", "hint": "Make sure the URL is public, uses http(s), and is not a private address."}}
```

| HTTP | code | Cause | Recovery |
|---|---|---|---|
| 422 | `skill_fetch_failed` | `skill_url` unreachable, non-200, or blocked as private | Check the URL in a `POST /v1/probe` call; fix or use `skill_markdown` |
| 422 | (FastAPI validation detail) | both or neither of `skill_url`/`skill_markdown` sent; an inline `skill_markdown` longer than 200000 chars; or any field over its length limit | Send exactly one source field, within the size limits |
| 413 | `skill_too_large` | a fetched `skill_url` returned a body larger than 200000 bytes | Shorten the hosted SKILL.md |
| 500 | `internal_error` | unexpected server error | Retry once; then retry with `"execute": false` |

### 3. GET /v1/reports/{report_id}

Purpose: re-fetch a report produced in the last 15 minutes.

Path parameter: `report_id` — the exact `report_id` string from a `/v1/validate` response.

Request:

```bash
curl -s https://practical-fulfillment-production-9e7f.up.railway.app/v1/reports/rep_1a2b3c4d5e6f7a8b
```

Response: the identical report object shown above (HTTP 200), or HTTP 404 with code `report_not_found` after expiry. Recovery for 404: re-run `POST /v1/validate` with the same input; the same `report_id` is regenerated.

### 4. POST /v1/probe

Purpose: safe reachability check for one public URL (GET or HEAD only). Use it to diagnose a failing service before or after a validation.

Body fields:

| Field | Type | Required | Constraints |
|---|---|---|---|
| `url` | string | yes | public http(s) URL, ≤ 2000 chars |
| `method` | string | no | `GET` (default) or `HEAD` |

Request:

```bash
curl -s -X POST https://practical-fulfillment-production-9e7f.up.railway.app/v1/probe \
  -H "Content-Type: application/json" \
  -d '{"url": "https://practical-fulfillment-production-9e7f.up.railway.app/health"}'
```

Response (HTTP 200, application/json):

```json
{"url": "https://example-skill.app/health", "final_url": "https://example-skill.app/health", "reachable": true, "status_code": 200, "latency_ms": 120, "content_type": "application/json", "redirects": 0, "error": null, "body_preview": "{\"status\":\"ok\"}", "limits": {"max_response_bytes": 262144}}
```

If `reachable` is `false`, the `error` field states why (`timeout`, `connection_error: ...`, `unsafe_url: ...`). Private and localhost URLs always return `reachable: false` with an `unsafe_url` error; this is intentional and not retryable.

### 5. GET /skill.md

Purpose: returns this document. Useful for self-validation demos.

```bash
curl -s https://practical-fulfillment-production-9e7f.up.railway.app/skill.md
```

## Step-by-step: how an agent validates a skill autonomously

1. Send `GET https://practical-fulfillment-production-9e7f.up.railway.app/health`. Expect `{"status": "ok"}`. If this fails, wait 30 seconds and retry once before reporting SkillProbe unavailable.
2. Send `POST https://practical-fulfillment-production-9e7f.up.railway.app/v1/validate` with a JSON body: set `skill_url` to the exact SKILL.md URL you were asked to check, and set `execute` to `true`. Do not invent a URL: if you were given Markdown text instead of a URL, put that text in `skill_markdown` instead.
3. Read `passed`. If `true`, the skill is safe to use; you are done.
4. If `passed` is `false`, collect every element of `checks` where `passed` is `false`. Each one's `hint` is the fix. Report these to your principal, or apply them if you own the skill, then re-validate.
5. If validation returned `skill_fetch_failed`, send `POST /v1/probe` with the SKILL.md URL to determine whether the file itself is unreachable, then report the probe's `error` field.

Success condition: you have a report object with a `report_id`, and you have either (a) `passed: true`, or (b) the complete list of failed checks with their hints. Either outcome completes the task.

## Complete end-to-end example

Task: "Check whether the skill documented at https://example-skill.app/skill.md is usable."

```bash
# 1. SkillProbe is alive?
curl -s https://practical-fulfillment-production-9e7f.up.railway.app/health
# -> {"status":"ok","service":"skillprobe","version":"1.0.0","time":1752192000}

# 2. Validate the target skill.
curl -s -X POST https://practical-fulfillment-production-9e7f.up.railway.app/v1/validate \
  -H "Content-Type: application/json" \
  -d '{"skill_url": "https://example-skill.app/skill.md", "execute": true}'
# -> {"report_id":"rep_...","score":92,"grade":"A","passed":true, ...}

# 3. Done: report passed=true, score, and any failed checks to your principal.
```

## Rules for the agent

- Never invent field values. If you do not have a SKILL.md URL or its text, ask for it or stop; do not fabricate one.
- Send exactly one of `skill_url` or `skill_markdown`, never both, never neither.
- Requests are idempotent: repeating an identical `/v1/validate` call is always safe and returns the cached report.
- Rate behavior: SkillProbe sends at most 10 outbound requests per validation and caps responses at 256 KiB; validations normally finish in under 15 seconds. If a validation takes longer than 30 seconds, retry once with `"execute": false` to get the static report.
- All timestamps are Unix seconds. All bodies are UTF-8 JSON unless documented otherwise.