简体中文 · English

📰 last30days-cn — In-Depth Research Engine for Chinese Platforms

🚀 30 days of research, in 30 seconds. 8 major platforms. No stale information.

last30days-cn is an AI Agent skill that automatically searches the content from the last 30 days across 8 major Chinese internet platforms, synthesizes the results, and generates well-supported research reports.

🔗 This project is a deeply localized adaptation based on mvanhorn/last30days-skill, fully tailored for Chinese users and Chinese internet platforms.

🕷️ v2.0 integrates the crawling engine approach from MediaCrawler, significantly reducing the dependency on API Keys. v2.1 fixed anti-crawler issues on Baidu and Xiaohongshu: XHR interception now replaces DOM parsing, Bing acts as a fallback search, and the ineffective ScrapeCreators Xiaohongshu integration has been removed.

Current version: v3.2.0

👤 Author: Jesse (@Jesseovo)

---

✨ v3.2.0 Improvements

• Fixed an issue where the new Xiaohongshu search page only triggered search/recommend autocomplete suggestions: instead of treating sug_items as notes, results are now identified from note cards in the response, and the search box is actively submitted when necessary.
• The Playwright crawler no longer depends on a fixed Xiaohongshu endpoint; as long as the response structure still contains note cards during platform updates, parsing works normally.
• Added a legacy computer compatibility mode: you can use LAST30DAYS_BROWSER_PATH to point to a Chromium/Chrome that your system still supports, or use LAST30DAYS_DISABLE_BROWSER=1 to fully disable the browser and fall back to public APIs/search.
• --diagnose now shows browser mode, external browser path, and whether the path exists, making it easier to troubleshoot browser binary compatibility on older macOS systems.
• Version bumped to v3.2.0; root-directory scripts and the Agent Skills installation payload continue to stay in sync via payload checks.

✨ v3.1.0 Improvements

• Fixed Chinese platform date normalization to Beijing Time (CST) archiving, avoiding window boundary offsets on machines not in Beijing Time.
• Added CJK bigram fallback tokenization; jieba is now an optional enhancement, and the skill can run with zero hard dependencies.
• Unified HTTP retry backoff, Retry-After parsing, and debug URL redaction.
• Added payload generation/drift checks, a single source of version truth, cache wiring, and GitHub Actions CI.

✨ v3.0.0 Upgrades

• Aligned with the upstream v3 Agent Skills package structure: skills/last30days is now a standalone installable runtime payload.
• The Chinese CLI now uses a single entry point last30days.py, with the root directory and Skill payload sharing the same structure.
• Added --emit html and --emit html-path, which can generate an offline-openable report.html.
• The HTML report incorporates the Swiss/IKB visual language from op7418/guizang-ppt-skill, suitable for browsing, archiving, and printing.
• Added empty-result fallback notes for Xiaohongshu and Zhihu searches; failures now annotate attempted paths and possible causes.
• Douyin and Toutiao now fall back to public search engines when native APIs are rate-limited, instead of silently returning 0 results (see issue #8).
• Fixed npx installation failures on macOS/Linux caused by corrupted symlinks in skills/last30days/SKILL.md (see issue #10).
• Synced several platform-agnostic features with upstream mvanhorn/last30days-skill: --as-of historical backtracking, cross-platform aggregated hot topics, LAST30DAYS_DEFAULT_SEARCH/EXCLUDE_SOURCES configuration switches, honest --diagnose (live probing), and HTML report XSS hardening.
• Root-directory scripts/ is retained for local development and legacy path calls; Agent Skills installation uses the self-contained payload under skills/last30days/scripts.

✅ Release Verification

Pre-release checks completed:

• The remote release tag is unified as v3.2.0, with no extra v3 derivative tags.
• Both the root directory and Skill payload use a single last30days.py entry point, with no extra entry files.
• Full test suite passed: python -m pytest tests -q, with 214 passed.
• Both the root entry and the Skill payload entry have been verified: py scripts/last30days.py --diagnose and py skills/last30days/scripts/last30days.py --diagnose both output platform availability diagnostics correctly.

---

⚠️ Disclaimer

Please read the following carefully. By using this project, you agree to all of the terms below.

Legal Compliance Statement

1. This project is for learning and research only. All crawler features are for technical learning and research exchange, not for commercial use.
2. Users must comply with relevant PRC laws, including but not limited to:
   • Cybersecurity Law of the PRC
   • Data Security Law of the PRC
   • Personal Information Protection Law of the PRC
   • Anti-Unfair Competition Law of the PRC
3. Users must comply with each platform's Terms of Service (ToS) and robots.txt.
4. Prohibited uses include:
   • Large-scale, high-frequency scraping of platform data
   • Collecting, storing, or disseminating others' personal information
   • Disrupting or interfering with platform operations
   • Any illegal data resale or commercial profiteering
   • Providing automated data collection services externally
5. The developer is not liable for any legal issues arising from use of this project. Users bear all legal risks themselves.
6. For infringement concerns, contact the author for prompt resolution.

Technical Disclaimer

• Crawler functionality uses Playwright browser automation to simulate normal browsing, and does not involve reverse engineering or bypassing security mechanisms.
• Platform APIs may change at any time; no guarantee of perpetual availability.
• Keep request frequency reasonable (e.g., ≥ 5s interval per search) to avoid being blocked.

💡 Crawler-related legal violations happen frequently; please ensure you use this in full legal compliance. Reference: Collection of Chinese crawler-related legal cases

---

✨ v2.0 New Features

🆕 v2.0 vs v1.0 Comparison

Feature	v1.0	v2.0
Platforms available for free	4	7 (after installing Playwright)
Platforms requiring an API Key	Weibo, Xiaohongshu, Douyin, WeChat	WeChat only (others can use the crawler instead)
Data acquisition methods	API + public endpoints only	API + crawler engine + public endpoints
Installation complexity	Requires configuring multiple API Keys	pip install playwright is enough
marketplace.json	Missing owner field (Bug)	✅ Fixed

Core Upgrades

1. Integrated MediaCrawler crawler engine — based on Playwright browser automation, no reverse engineering of encryption algorithms required, greatly lowering the barrier to entry
2. 7 of 8 platforms available with zero configuration — except for WeChat, all platforms can be used without an API Key
3. Smart fallback strategy — API first → crawler mode → public endpoints, three-tier automatic fallback
4. Fixed Issue #1 — fixed the bug where the missing owner field in marketplace.json caused Claude Code installation to fail
5. Login session caching — crawler mode supports Cookie persistence, reducing repeated logins

---

📋 Platform Support

Platform	Module	Data Acquisition Method	Configuration Required
🔴 Weibo	weibo.py	API / 🕷️ Crawler / Public endpoints	✅ No config needed for crawler mode
📕 Xiaohongshu	xiaohongshu.py	API / 🕷️ Crawler / Public endpoints / Bing fallback	✅ No config needed for crawler mode
📺 Bilibili	bilibili.py	Public API / 🕷️ Crawler backup	✅ No config needed
💬 Zhihu	zhihu.py	Public search / 🕷️ Crawler backup	✅ No config needed
🎵 Douyin	douyin.py	API / 🕷️ Crawler / Public endpoints / Search fallback	✅ No config needed for crawler mode
💚 WeChat	wechat.py	API / Sogou search	WECHAT_API_KEY (optional)
🔵 Baidu	baidu.py	Public search / API	✅ Basic search requires no config
📰 Toutiao	toutiao.py	Public endpoints / Search fallback	✅ No config needed

🕷️ = optional Playwright browser mode (pip install playwright && playwright install chromium); on older systems you can use LAST30DAYS_BROWSER_PATH or rely on public search fallback.

---

🤖 Agent Platform Installation

Agent Skills (Recommended)

npx skills add Jesseovo/last30days-skill-cn -g

Cursor (Recommended)

Clone the project into the Cursor skills directory:

git clone https://github.com/Jesseovo/last30days-skill-cn.git

Then add SKILL.md as a project skill in Cursor.

Claude Code

# 方式一：通过 Agent Skills 安装（推荐）
npx skills add Jesseovo/last30days-skill-cn -g

# 方式二：手动安装
git clone https://github.com/Jesseovo/last30days-skill-cn.git ~/.claude/skills/last30days-cn

OpenClaw / ClawHub

git clone https://github.com/Jesseovo/last30days-skill-cn.git ~/.agents/skills/last30days-cn

Gemini CLI

git clone https://github.com/Jesseovo/last30days-skill-cn.git
# 在 Gemini CLI 中作为扩展加载

Generic Agent

Any AI Agent that supports Bash / Read / Write tools can use this skill.

---

⚙️ Configuration Guide

📍 Step 1: Optional Enhancements

python -m pip install jieba

jieba is not a hard dependency; when not installed, it automatically falls back to CJK bigram tokenization and still runs.

📍 Step 2: Install the Crawler Engine (Optional, Recommended — enables 7/8 platforms)

python -m pip install playwright
python -m playwright install chromium

After installing Playwright, Weibo, Xiaohongshu, Douyin, Bilibili (backup), and Zhihu (backup) can all be used without an API Key.

Legacy macOS / Older Computer Compatibility Mode

Playwright-managed Chromium raises system requirements with each version. When older systems like macOS Catalina cannot launch the newer browser, you don't have to abandon the entire skill:

1. Continue using public APIs and Bing site search fallback (without installing Playwright), or install a Chromium/Chrome build that still supports your system.
2. Point Playwright at a local browser executable:

export LAST30DAYS_BROWSER_PATH="/Applications/Chromium.app/Contents/MacOS/Chromium"
python scripts/last30days.py --diagnose

You can also use an installed Chrome channel:

export LAST30DAYS_BROWSER_CHANNEL=chrome

To force skipping all browser invocations:

export LAST30DAYS_DISABLE_BROWSER=1
python scripts/last30days.py "AI 工具" --search weibo,xiaohongshu,bilibili,zhihu,douyin,baidu,toutiao

LAST30DAYS_BROWSER_PATH must point to a browser that can actually launch on your older system; the skill will not download or upgrade your system browser. Browser-based crawling is still affected by platform login state, CAPTCHAs, and API revisions; Xiaohongshu's search/recommend related-term suggestions are not counted in search results.

📍 Step 3: Create Configuration File (Optional)

If you'd like to use API mode for more stable data, or need WeChat Official Account search:

mkdir -p ~/.config/last30days-cn
touch ~/.config/last30days-cn/.env
chmod 600 ~/.config/last30days-cn/.env

Equivalent in Windows (PowerShell): you can create the directory and empty config file with New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\last30days-cn" and New-Item -ItemType File -Path "$env:USERPROFILE\.config\last30days-cn\.env" -Force. To restrict .env to read/write by the current user only, you can approximate with icacls "$env:USERPROFILE\.config\last30days-cn\.env" /inheritance:r /grant:r "$($env:USERNAME):(R,W)" (similar intent to Unix chmod 600, though the permission model differs).

Edit ~/.config/last30days-cn/.env and fill in API Keys as needed:

# ============================================
# last30days-cn v2.0 配置文件
# ============================================
# 📌 说明：所有 API Key 均为可选
# 安装 Playwright 后，大部分平台已可通过爬虫模式使用
# API Key 提供更稳定的数据获取方式
# ============================================

# 🔴 微博开放平台（可选，已有爬虫模式替代）
# 获取方式: https://open.weibo.com → 创建应用 → 获取 Access Token
WEIBO_ACCESS_TOKEN=

# 📕 小红书（可选，已有爬虫模式替代）
# 获取方式: https://scrapecreators.com → 注册 → 获取 API Key
SCRAPECREATORS_API_KEY=

# 💬 知乎 Cookie（可选，增强搜索质量）
# 获取方式: 浏览器登录知乎 → F12 → Network → 复制 Cookie 值
ZHIHU_COOKIE=

# 🎵 抖音（可选，已有爬虫模式替代）
# 获取方式: https://tikhub.io → 注册 → 获取 API Key
TIKHUB_API_KEY=

# 💚 微信公众号搜索（目前无爬虫替代，需 API Key 才能使用）
# 获取方式: 使用第三方微信搜索 API 服务商
WECHAT_API_KEY=

# 🔵 百度搜索 API（可选，公开搜索已可用）
# 获取方式: https://cloud.baidu.com → 搜索服务 → 创建应用
BAIDU_API_KEY=
BAIDU_SECRET_KEY=

📍 Step 4: Verify Configuration

python scripts/last30days.py --diagnose

Displays availability and crawler status:

{
  "weibo": true,
  "xiaohongshu": false,
  "bilibili": true,
  "zhihu": true,
  "douyin": true,
  "wechat": false,
  "baidu_api": false,
  "toutiao": true,
  "crawler_engine": {
    "playwright_available": true,
    "cached_logins": [],
    "note": "安装 Playwright 后，微博/小红书/抖音/B站/知乎可无需 API Key 使用爬虫模式"
  },
  "note_douyin_toutiao": "抖音/头条原生接口需签名参数，常被风控；接口失败时改用公开搜索引擎兜底，仅能拿到公开链接，无真实互动数据与精确日期。"
}

---

🚀 Usage

Basic Usage

python scripts/last30days.py "AI编程助手" --emit compact
python scripts/last30days.py "AI编程助手" --emit html-path

Command-line Options

Option	Description	Example
--emit	Output mode	compact / json / md / context / path / html / html-path
--quick	Quick search	Fewer data sources, faster speed
--deep	Deep search	More data sources, more comprehensive
--days N	Lookback days	--days 7 (last week)
--as-of	Historical lookback end date	--as-of 2026-05-01 (look back N days ending on that date)
--search	Specify search sources	--search weibo,bilibili,zhihu
--diagnose	Diagnose configuration	Show availability status of each platform
--timeout SECS	Global timeout in seconds	Overrides the default global timeout
--save-dir DIR	Auto-save raw output directory	Write raw output to the specified directory
--debug	Debug mode	Output detailed logs

🔧 Environment variables: when --search is not specified, falls back to LAST30DAYS_DEFAULT_SEARCH (comma-separated default source set); EXCLUDE_SOURCES can exclude specified sources from the enabled set.

Usage Examples

# 🔍 搜索 AI 相关话题
python scripts/last30days.py "最新AI工具" --emit compact

# ⚡ 快速搜索，仅 B站和知乎
python scripts/last30days.py "Python教程" --quick --search bilibili,zhihu

# 📊 深度搜索并保存结果
python scripts/last30days.py "新能源汽车" --deep --save-dir ~/Documents/research

# 📋 输出 JSON 格式（适合程序处理）
python scripts/last30days.py "ChatGPT替代品" --emit json

# 🗓️ 仅搜索最近 7 天
python scripts/last30days.py "热门话题" --days 7

# 🧾 生成可离线打开的 HTML 报告
python scripts/last30days.py "具身智能" --deep --emit html-path

---

🔧 Data Acquisition Strategy (Three-Tier Fallback)

v2.0 uses a three-tier automatic fallback to maximize availability:

优先级 1: API 模式（如配置了 API Key）
    ↓ 失败或未配置
优先级 2: 爬虫模式（MediaCrawler，需要 Playwright）
    ↓ 失败或未安装
优先级 3: 公开接口（HTTP 直接请求，无需任何配置）
    ↓ 仍无结果（抖音/头条/小红书/知乎）
兜底: 公开搜索引擎（Bing site: 搜索，获取公开链接）

Data Acquisition Comparison by Platform

Platform	API Mode	Crawler Mode	Public Endpoints	Search Fallback
Weibo	WEIBO_ACCESS_TOKEN	✅ Playwright	✅ m.weibo.cn	-
Xiaohongshu	MCP HTTP API (optional)	✅ Playwright (note payload recognition)	⚠️ Low hit rate	✅ Bing
Bilibili	-	✅ Playwright (backup)	✅ Public API	-
Zhihu	ZHIHU_COOKIE (enhanced)	✅ Playwright (backup)	✅ Public search	✅ Bing
Douyin	TIKHUB_API_KEY	✅ Playwright	⚠️ Signature required	✅ Bing
WeChat	WECHAT_API_KEY	-	✅ Sogou search	-
Baidu	BAIDU_API_KEY	-	⚠️ Public search may be blocked	✅ Bing
Toutiao	-	-	⚠️ Signature required	✅ Bing

⚠️ Douyin/Toutiao native web endpoints now require signature parameters (a_bogus / _signature), which are often risk-controlled. When the API fails, fall back to public search engines, but you can only get public links, no real engagement data or precise dates.

---

🏗️ Project Architecture

last30days-skill-cn/
├── 📄 SKILL.md              # Agent 技能定义文件
├── 📄 README.md             # 项目说明（中文，本文件）
├── 📄 README.en.md          # 项目说明（English）
├── 📄 LICENSE               # MIT 许可证
├── 📄 requirements.txt      # Python 依赖
├── 📁 assets/               # README 配图
├── 📁 scripts/
│   ├── 🐍 last30days.py     # 中文主入口 CLI
│   └── 📁 lib/
│       ├── crawler_bridge.py  # 🆕 MediaCrawler 爬虫桥接模块
│       ├── weibo.py          # 微博搜索模块
│       ├── xiaohongshu.py    # 小红书搜索模块
│       ├── bilibili.py       # B站搜索模块
│       ├── zhihu.py          # 知乎搜索模块
│       ├── douyin.py         # 抖音搜索模块
│       ├── wechat.py         # 微信公众号模块
│       ├── baidu.py          # 百度搜索模块
│       ├── toutiao.py        # 今日头条模块
│       ├── schema.py         # 数据结构定义
│       ├── score.py          # 评分系统
│       ├── normalize.py      # 数据标准化
│       ├── dedupe.py         # 去重
│       ├── render.py         # 输出渲染
│       ├── relevance.py      # 相关性计算
│       ├── query.py          # 查询预处理
│       ├── query_type.py     # 查询类型检测
│       ├── entity_extract.py # 实体抽取
│       ├── env.py            # 环境配置管理
│       ├── cache.py          # 缓存管理
│       ├── dates.py          # 日期工具
│       ├── http.py           # HTTP 客户端
│       ├── ui.py             # 终端 UI
│       └── setup_wizard.py   # 配置向导
├── 📁 skills/
│   └── 📁 last30days/        # Agent Skills 自包含运行载荷
│       ├── 📄 SKILL.md       # 安装后供 Agent 读取的技能说明
│       └── 📁 scripts/
│           ├── 🐍 last30days.py
│           └── 📁 lib/       # 与根目录脚本同步的运行依赖
├── 📁 fixtures/              # 示例数据
├── 📁 tests/                 # 测试用例
└── 📁 hooks/                 # Agent 钩子

---

📊 Scoring System

The composite score (0–100) of each search result is based on:

Dimension	Weight	Description
🎯 Relevance	45%	Textual match with the query topic
🕐 Recency	25%	Freshness of the content publication time
🔥 Engagement	30%	Platform-specific engagement metrics (see table below)

Platform Metrics

Platform	Engagement Metrics
Weibo	Reposts + Comments + Likes
Xiaohongshu	Likes + Favorites + Comments + Shares
Bilibili	Plays + Danmaku + Comments + Coins + Favorites
Zhihu	Agrees + Comments + Favorites
Douyin	Likes + Comments + Shares + Plays
Toutiao	Comments + Views + Likes

---

🙏 Acknowledgments

• mvanhorn/last30days-skill — original English project
• NanmiCoder/MediaCrawler — technical inspiration for the crawler engine

---

📜 License

This project is released under the MIT License.

• 🔗 Original project: mvanhorn/last30days-skill by Matt Van Horn
• 🇨🇳 Chinese localization: Jesse (@Jesseovo)