Commit Graph

312 Commits

Author SHA1 Message Date
github-actions[bot] edfac9f24c style: auto format 2025-11-27 05:54:33 +00:00
github-actions[bot] 838eb3c7ff style: auto format 2025-11-27 05:49:33 +00:00
github-actions[bot] a28890a135 style: auto format 2025-11-27 02:55:46 +00:00
Tony 2ae10a7a02
feat: replace rand-user-agent with header-generator (#20480)
* Initial plan

* Replace @tonyrl/rand-user-agent with header-generator and add sec-ch-* headers

Co-authored-by: TonyRL <11386903+TonyRL@users.noreply.github.com>

* Extract unwanted UA validation into standalone isValidUserAgent function

Co-authored-by: TonyRL <11386903+TonyRL@users.noreply.github.com>

* Add headerGeneratorPreset option to got and ofetch utilities

* Replace randUserAgent usage in lib/routes with headerGeneratorPreset option

Co-authored-by: TonyRL <11386903+TonyRL@users.noreply.github.com>

* chore: pin header-generator to exact match

* fix: replace randUserAgent with a static user agent string in config

* Remove deprecated _randUserAgent function and related tests

Co-authored-by: TonyRL <11386903+TonyRL@users.noreply.github.com>

* refactor: rename to header-generator

* Changes before error encountered

Co-authored-by: TonyRL <11386903+TonyRL@users.noreply.github.com>

* fix: update generateHeaders parameter type to Partial<HeaderGeneratorOptions>

* fix: add ios preset

* fix: rename headerGeneratorPreset to headerGeneratorOptions

* feat: add MODERN_IOS preset and update header-generator patch hash

* fix: use ios preset

* fix: update FetchOptions to include headerGeneratorOptions type

* refactor: simplify header generation and update header inclusion list

* fix: remove header-generator patch

* refactor: update user-agent handling in requests and tests

* test: remove redundant headerGeneratorOptions test and add new ofetch header preset test

* test: add custom user-agent test for ofetch

* test: update user-agent expectation in ofetch headers test

* Update lib/utils/ofetch.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor: replace hardcoded headers with generatedHeaders in fetch and get request handlers

* refactor: rename test suite from rand-user-agent to header-generator and remove unused dependency

* Update lib/utils/header-generator.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: TonyRL <11386903+TonyRL@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-14 00:49:41 +08:00
dependabot[bot] 52c18b7f25
chore(deps-dev): bump eslint-plugin-unicorn from 61.0.2 to 62.0.0 (#20381)
* chore(deps-dev): bump eslint-plugin-unicorn from 61.0.2 to 62.0.0

Bumps [eslint-plugin-unicorn](https://github.com/sindresorhus/eslint-plugin-unicorn) from 61.0.2 to 62.0.0.
- [Release notes](https://github.com/sindresorhus/eslint-plugin-unicorn/releases)
- [Commits](https://github.com/sindresorhus/eslint-plugin-unicorn/compare/v61.0.2...v62.0.0)

---
updated-dependencies:
- dependency-name: eslint-plugin-unicorn
  dependency-version: 62.0.0
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* style: fix unicorn/prefer-string-raw

* style: fix unicorn/no-useless-collection-argument

* style: fix unicorn/no-immediate-mutation

* style: fix unicorn/no-immediate-mutation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-28 01:26:56 +08:00
Stephen Zhou f8c1415b8e
fix(route/bilibili): duration for video route 2025-10-09 10:44:18 +08:00
Tony 49685ad6b3
fix(ofetch): include error cause in request error logging (#20225) 2025-10-04 04:50:53 +08:00
dependabot[bot] 802cbf95d2
chore(deps-dev): bump eslint-plugin-unicorn from 60.0.0 to 61.0.2 (#19998)
* chore(deps-dev): bump eslint-plugin-unicorn from 60.0.0 to 61.0.1

Bumps [eslint-plugin-unicorn](https://github.com/sindresorhus/eslint-plugin-unicorn) from 60.0.0 to 61.0.1.
- [Release notes](https://github.com/sindresorhus/eslint-plugin-unicorn/releases)
- [Commits](https://github.com/sindresorhus/eslint-plugin-unicorn/compare/v60.0.0...v61.0.1)

---
updated-dependencies:
- dependency-name: eslint-plugin-unicorn
  dependency-version: 61.0.1
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* refactor: replace `replace(/regex/g)` with replaceAll

* refactor: replace sort() with toSorted() for immutability in various routes and utilities

* refactor: fix unicorn/prefer-array-find

* refactor: fix unicorn/prefer-at

* refactor: add unicorn/no-array-sort rule to ESLint configuration

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-08 23:03:48 +08:00
DIYgod 532619a8c2
fix(tests): update expected values and increase timeout for Puppeteer tests
- Adjusted expected values in error handling tests to reflect updated application behavior.
- Increased timeout for Puppeteer tests from 20 seconds to 45 seconds to accommodate longer loading times.
2025-08-22 17:45:39 +08:00
DIYgod bbe768246f
feat: vercel compatibility 2025-08-11 21:39:03 +08:00
dependabot[bot] ae3e646ea9
chore(deps-dev): bump eslint-plugin-unicorn from 59.0.1 to 60.0.0 (#19662)
* chore(deps-dev): bump eslint-plugin-unicorn from 59.0.1 to 60.0.0

---
updated-dependencies:
- dependency-name: eslint-plugin-unicorn
  dependency-version: 60.0.0
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* refactor: replace reverse() with toReversed() for improved readability

* chore: update packageManager version to pnpm@10.13.1

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-22 20:45:42 +08:00
Tony 857d5989c1
feat(utils/puppeteer): replace `puppeteer-extra-plugin-stealth` with `rebrowser` (#19506)
* feat: use rebrowser

* fix: remove rebrowser-puppeteer-core

* fix: chrome not found

* fix: replace remove-infobar with disable-blink-features

remove-infobar was removed since M76 https://github.com/puppeteer/puppeteer/issues/1765

* feat: remove outdated stealth plugin

* fix: remove puppeteer non stealth test
2025-07-01 04:38:05 +08:00
pseudoyu 3d9ac5c6ee
feat(lib/proxy): support multiple proxier by `PROXY_URIS` env 2025-06-24 11:05:14 +08:00
github-actions[bot] 8abc00fe2e style: auto format 2025-06-22 16:44:52 +00:00
夜阑听风 90eb7bbacf
feat: support float score threshold for douban (#19425)
* support float score threshold for douban

* add readable-social tests
2025-06-23 00:42:44 +08:00
纪华裕 401c0c563f
feat(cache): enhance tryGet method to support generic return types (#19261) 2025-06-03 09:37:43 +08:00
Tony 5cf663659a fix: add 'networkidle2' option to waitUntil in getPuppeteerPage 2025-05-29 04:02:55 +08:00
DIYgod cf8e3bdf45
feat: bilibili new puppeteer 2025-05-28 22:07:44 +08:00
DIYgod 27305f107a
test: fix puppeteer 2025-05-28 21:26:06 +08:00
DIYgod bf3649324b
fix: puppeteer proxy regex 2025-05-28 21:12:21 +08:00
DIYgod 34c43eaa0d
fix: puppeteer proxy 2025-05-28 12:51:43 +08:00
DIYgod 502daae0c1
feat: video duration 2025-05-18 11:54:06 +08:00
Tony 8faec7fbfd refactor: fix Element type import 2025-05-07 04:18:23 +08:00
Tony f68f81c49d
fix(utils/common-config): import parseDate and timezone functions for date handling (#19023) 2025-05-06 02:07:35 +08:00
Tony a5e2e295dc
refactor: fix all `get` to `toArray` (#19000)
* refactor: fix all get

* fix(utils/common-config): charset case compare

* Revert "fix(utils/common-config): charset case compare"

This reverts commit 3c1880f2fad53ad2cafddcee7741390fd46fa3d2.

* fix: nga

* fix: notateslaapp

* fix: ps

* fix: xjtu

* fix: enforce error on restricted syntax for get calls

* fix: eslint

* fix: more eslint
2025-05-04 05:24:53 +08:00
Tony f19a725157
refactor: fix all no array reduce (#18973)
* refactor: no-array-reduce cleanup

* refactor: keep

* refactor: nhk

* refactor: rattibha

* fix: cctv

* fix: jiemian

* refactor: juejin

* refactor: kpopping

* refactor: yoasobi-music

* refactor: mycard520

* refactor: newrank

* refactor: sse

* refactor: zhihu

* feat!: remove trending

https://github.com/huqi-pr/trending-in-one is archived and no longer being updated.
2ef4cd236f

* refactor: sspai

* refactor: sysu

* refactor: nikkei

* refactor: utils

* refactor: mydrivers

* refactor: scmp

* refactor: gov/moa

* refactor: bloomberg

* refactor: manhuagui

* refactor: twitter
2025-04-30 04:16:32 +08:00
Tony 03969f3867
refactor: fix lint issue (#18963)
* fix: unicorn/no-empty-file

* fix: unicorn/prefer-node-protocol

* fix: unicorn/prefer-number-properties

* fix: unicorn/consistent-destructuring

* fix: unicorn/no-array-callback-reference

* fix: unicorn/new-for-builtins

* fix: set checkNaN to false in unicorn/prefer-number-properties

* fix: unicorn/prefer-string-slice

* fix: update eslint ignore
2025-04-29 03:00:16 +08:00
DIYgod 83e701b592
feat: compile ts through tsdown (#18908)
* feat: build ts via tsdown

* feat: exec tsdown

* fix

* fix

* fix

* feat: use tsdown shims

* chore: run rolldown lifecycle script

* fix: use shims

---------
2025-04-22 22:29:11 +08:00
dependabot[bot] c37c80f38d
chore(deps): bump the opentelemetry group with 5 updates (#18637)
* chore(deps): bump the opentelemetry group with 5 updates

Bumps the opentelemetry group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [@opentelemetry/exporter-prometheus](https://github.com/open-telemetry/opentelemetry-js) | `0.57.2` | `0.200.0` |
| [@opentelemetry/exporter-trace-otlp-http](https://github.com/open-telemetry/opentelemetry-js) | `0.57.2` | `0.200.0` |
| [@opentelemetry/resources](https://github.com/open-telemetry/opentelemetry-js) | `1.30.1` | `2.0.0` |
| [@opentelemetry/sdk-metrics](https://github.com/open-telemetry/opentelemetry-js) | `1.30.1` | `2.0.0` |
| [@opentelemetry/sdk-trace-base](https://github.com/open-telemetry/opentelemetry-js) | `1.30.1` | `2.0.0` |


Updates `@opentelemetry/exporter-prometheus` from 0.57.2 to 0.200.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-js/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-js/compare/experimental/v0.57.2...experimental/v0.200.0)

Updates `@opentelemetry/exporter-trace-otlp-http` from 0.57.2 to 0.200.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-js/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-js/compare/experimental/v0.57.2...experimental/v0.200.0)

Updates `@opentelemetry/resources` from 1.30.1 to 2.0.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-js/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-js/compare/v1.30.1...v2.0.0)

Updates `@opentelemetry/sdk-metrics` from 1.30.1 to 2.0.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-js/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-js/compare/v1.30.1...v2.0.0)

Updates `@opentelemetry/sdk-trace-base` from 1.30.1 to 2.0.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-js/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-js/compare/v1.30.1...v2.0.0)

---
updated-dependencies:
- dependency-name: "@opentelemetry/exporter-prometheus"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: opentelemetry
- dependency-name: "@opentelemetry/exporter-trace-otlp-http"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: opentelemetry
- dependency-name: "@opentelemetry/resources"
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: opentelemetry
- dependency-name: "@opentelemetry/sdk-metrics"
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: opentelemetry
- dependency-name: "@opentelemetry/sdk-trace-base"
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: opentelemetry
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(otel): upgrade to otel v2

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
2025-03-25 18:54:32 +08:00
Tony 674b32262a
fix: ensure proper browser closure (#18404) 2025-02-20 04:04:46 +08:00
dependabot[bot] 8a0efc0ba2
chore(deps-dev): bump eslint-plugin-unicorn from 56.0.1 to 57.0.0 (#18392)
* chore(deps-dev): bump eslint-plugin-unicorn from 56.0.1 to 57.0.0

Bumps [eslint-plugin-unicorn](https://github.com/sindresorhus/eslint-plugin-unicorn) from 56.0.1 to 57.0.0.
- [Release notes](https://github.com/sindresorhus/eslint-plugin-unicorn/releases)
- [Commits](https://github.com/sindresorhus/eslint-plugin-unicorn/compare/v56.0.1...v57.0.0)

---
updated-dependencies:
- dependency-name: eslint-plugin-unicorn
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: remove unicorn plugin

* style: fix eslint issues

* style: fix more issues

* chore: flat config unicorn

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
2025-02-19 05:10:11 +08:00
pseudoyu 3380b56488
fix(route/newrank): wechat route error 2024-11-29 19:25:26 +08:00
Tony 4e404a7328
chore: disable remote debugger by default (#17498) 2024-11-08 04:08:05 +08:00
bugyaluwang ff1ff8d940
feat(dev): visualize network requests in the development environment on Chrome Devtools's network tab (#17308)
* chore: add node-network-devtools to enhance development experience

* feat(dev): visualize network requests in the development environment on Chrome Devtools's network tab

* chore(deps): update node-network-devtools

* fix: lock error when ci

* chore: update node-network-devtools to v1.0.18

* chore: update node-network-devtools

* chore: use node-network-devtools v1.0.20

* feat: use hook to add custom headers for fetch

* fix: complex method 16

* test: add test for fetch

* fix: test type error

* fix: hard code

* chore(cr): rename the fetch.spec.ts to fetch.test.ts and use exact version

* chore: fix lock error

---------
2024-11-06 04:50:43 +08:00
Stephen Zhou e7526eada4
fix: get SearchParams string (#17445)
* fix: get SearchParams string

* refactor: use ufo

* lock deps
2024-11-05 11:27:13 +08:00
Stephen Zhou 4ffe081eb2
fix(bilibili): video reply (#17440)
* fix(bilibili): video reply

* fix

* update
2024-11-05 00:27:55 +08:00
Tony c97564446c
feat: log redirect (#17307) 2024-10-25 23:48:32 +08:00
dependabot[bot] edce496ba0
chore(deps): bump @opentelemetry/exporter-prometheus from 0.53.0 to 0.54.0 (#17286)
* chore(deps): bump @opentelemetry/exporter-prometheus

Bumps [@opentelemetry/exporter-prometheus](https://github.com/open-telemetry/opentelemetry-js) from 0.53.0 to 0.54.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-js/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-js/compare/experimental/v0.53.0...experimental/v0.54.0)

---
updated-dependencies:
- dependency-name: "@opentelemetry/exporter-prometheus"
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: fix pnpm install

* fix: use ATTR_SERVICE_NAME

SEMRESATTRS_SERVICE_NAME is deprecated

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-24 22:28:25 +08:00
Tony 2a1af7dc90
fix(cache): store cache TTL (#17044) 2024-10-09 00:37:02 +08:00
DIYgod 9b144a715f
fix(twitter): debug 2024-10-07 01:50:26 +08:00
DIYgod 3a9e9914d8
feat: increase maxQueueSize 2024-09-26 15:23:12 +08:00
Tony 62d72837b9 revert: "fix(common-config): add back parseDate and timezone"
This reverts commit 02efe692b2.

looks like tsx doesn't allow this
2024-09-20 11:21:30 +08:00
Tony 02efe692b2 fix(common-config): add bacl parseDate and timezone 2024-09-19 23:25:37 +08:00
incubator4 be3ba61788
chore: otel fix and enhancement (#16754)
* feat(otel): use routePath insteadof path

* feat(otel): add otel bucket boundary env

* chore: move otel to debug block with alphabetical sort
2024-09-15 19:42:21 +08:00
DIYgod 49df5559a1
feat: support ipv6, close #16513 2024-08-26 16:52:35 +08:00
DIYgod 50409ec19a
Revert "feat: puppeteer timeout"
This reverts commit 6d1a19a9bd.
2024-08-23 17:39:57 +08:00
DIYgod c1f8a74602
Revert "test: fix puppeteer timeout"
This reverts commit bbbe60dd5a.
2024-08-23 17:39:49 +08:00
DIYgod bbbe60dd5a
test: fix puppeteer timeout 2024-08-23 16:14:28 +08:00
DIYgod 6d1a19a9bd
feat: puppeteer timeout 2024-08-23 16:08:35 +08:00
Innei 2d82ddcddc
feat(route/rss3): RSS3 metadata parser (#16486)
* init

* done

Signed-off-by: Innei <i@innei.in>

* chore: cleanup

Signed-off-by: Innei <i@innei.in>

* fix: format

Signed-off-by: Innei <i@innei.in>

* fix: pass through

Signed-off-by: Innei <i@innei.in>

* chore: add testcase

Signed-off-by: Innei <i@innei.in>

* fix: update

Signed-off-by: Innei <i@innei.in>

---------

Signed-off-by: Innei <i@innei.in>
2024-08-21 22:49:59 +08:00