6d901cb7dc
style(eslint): add eslint-unicorn ( #14257 )
...
* style: add eslint-unicorn
* style: fix unicorn/no-useless-spread
* style: fix unicorn/no-useless-promise-resolve-reject
* style: fix unicorn/no-for-loop
* fix: codeql bad HTML filtering regexp
* fix: codeql incomplete replace
* fix: unicorn/no-abusive-eslint-disable
* style: fix unicorn/no-new-array
* style: fix unicorn/no-typeof-undefined
* style: fix unicorn/no-zero-fractions
* style: fix unicorn/no-empty-file
* style: fix unicorn/prefer-date-now
* revert: auto fix unicorn/prefer-switch on lib/v2/kuaidi100/utils.js
* style: fix unicorn/prefer-array-find
* style: fix unicorn/prefer-array-flat
* style: fix unicorn/prefer-array-flat-map
* style: fix unicorn/prefer-at
* style: fix unicorn/prefer-string-starts-ends-with
* style: fix unicorn/prefer-includes
* fix: codeql URL substring sanitization
* style: fix unicorn/prefer-optional-catch-binding
* style: fix unicorn/catch-error-name
* style: fix unicorn/escape-case
* style: fix unicorn/prefer-native-coercion-functions
* style: fix unicorn/prefer-regexp-test
* style: fix unicorn/require-array-join-separator
* style: fix unicorn/prefer-math-trunc
* style: fix unicorn/prefer-negative-index
* style: fix unicorn/prefer-dom-node-dataset
* style: fix unicorn/prefer-dom-node-text-content
* style: fix unicorn/prefer-query-selector
* style: fix unicorn/no-array-for-each
* style: fix unicorn/no-negated-condition
* style: fix unicorn/prefer-add-event-listener
* style: fix unicorn/import-style
* style: fix prefer-regex-literals
* style: disable unicorn/no-useless-switch-case
* style: disable unicorn/text-encoding-identifier-case
* style: fix unicorn/prefer-set-has
* style: fix unicorn/prefer-spread
* revert: auto fix on lib/routes/universities/ynnu/edu/base64.js
* style: fix unicorn/no-useless-undefined
* style: fix unicorn/no-array-push-push
* style: fix unicorn/no-useless-undefined again
* style: fix unicorn/no-lonely-if
* style: fix unicorn/prefer-reflect-apply
* style: fix unicorn/switch-case-braces
* style: fix unicorn/prefer-switch
* style: fix unicorn/prefer-array-some
* fix: deepscan UNUSED_VAR_ASSIGN
* style: fix unicorn/prefer-ternary
* fix: follow-up of unicorn/prefer-ternary
* revert: auto fix of unicorn/prefer-string-slice for substring()
* style: disable unicorn/prefer-string-slice
fix: auto fix slice over deprecated substr
* style: fix unicorn/throw-new-error
* style: fix unicorn/filename-case
* test: fix dateParser renaming
* style: fix unicorn/better-regex
* style: fix unicorn/prefer-string-replace-all
* fix(deps): add sanitize-html
* style: fix no-prototype-builtins
* style: fix unicorn/consistent-destructuring
* style: fix unicorn/consistent-function-scoping
* style: fix unicorn/prefer-regexp-test
* style: fix unicorn/prefer-logical-operator-over-ternary
* style: fix unicorn/no-array-callback-reference
* style: add prefer-object-has-own
* style: warn unicorn/no-empty-file
* style: fix unicorn/prefer-number-properties
* style: fix no-useless-undefined again
* style: fix unicorn/numeric-separators-style
* style: disable unicorn/no-array-callback-reference
false postive with cheerio
2024-01-18 20:43:40 +08:00
0e70524248
feat(proxy): add PAC script support ( #14218 )
...
* feat(proxy): add PAC script support
* fix pnpm fail
* fix coverage
* fix coverage final
* update docs
2024-01-13 02:35:56 +00:00
a1644ac4ca
feat: title case following The Chicago Manual of Style ( #14048 )
2023-12-15 22:37:12 +08:00
7da162041f
fix(core): duplicated path if the error is thrown from parameter ( #13967 )
...
* fix(core): double namespace if the error is thrown from parameter
* test: fix test case
2023-12-06 08:00:00 +08:00
eb1a40aa88
chore: fix random ua generation ( #13356 )
2023-09-22 03:32:48 +08:00
3a5d293b7f
feat: support reverse proxy
2023-07-23 20:28:38 +01:00
b982947a60
chore(deps): bump https-proxy-agent from 5.0.1 to 6.1.0 and socks-proxy-agent from 7.0.0 to 8.0.1 ( #12445 )
...
* chore(deps): bump https-proxy-agent from 5.0.1 to 6.0.0
Bumps [https-proxy-agent](https://github.com/TooTallNate/proxy-agents/tree/HEAD/packages/https-proxy-agent ) from 5.0.1 to 6.0.0.
- [Release notes](https://github.com/TooTallNate/proxy-agents/releases )
- [Changelog](https://github.com/TooTallNate/proxy-agents/blob/main/packages/https-proxy-agent/CHANGELOG.md )
- [Commits](https://github.com/TooTallNate/proxy-agents/commits/HEAD/packages/https-proxy-agent )
---
updated-dependencies:
- dependency-name: https-proxy-agent
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump https-proxy-agent from 5.0.1 to 6.0.0
* test: update test to use WHATWG URL API
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-07 00:25:12 +08:00
5b6d7cb524
style(eslint): add eslint-plugin-n ( #12393 )
2023-04-27 05:16:11 +08:00
3ff763ee8d
chore(deps): bump puppeteer from 19.8.3 to 19.11.0 ( #12385 )
...
* chore(deps): bump puppeteer from 19.8.3 to 19.11.0
Bumps [puppeteer](https://github.com/puppeteer/puppeteer ) from 19.8.3 to 19.11.0.
- [Release notes](https://github.com/puppeteer/puppeteer/releases )
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/release-please-config.json )
- [Commits](https://github.com/puppeteer/puppeteer/compare/puppeteer-v19.8.3...puppeteer-v19.11.0 )
---
updated-dependencies:
- dependency-name: puppeteer
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* test: fix puppeteer exitCode
* feat: use new headless chrome
https://developer.chrome.com/articles/new-headless/
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-26 03:14:45 +08:00
1373acf94c
test: use httpbingo for ua ( #12227 )
...
* test: use httpbingo for ua
* fix: lazy require maintainer
* test: extend timeout
2023-04-04 02:14:16 +08:00
6002eb6e11
test: test puppeteer-utils using `httpbingo.org` ( #11987 )
2023-03-01 02:40:30 +08:00
4c51a2b6c9
chore: skip stealth test in github actions
...
ref: https://docs.github.com/en/actions/learn-github-actions/variables#default-environment-variables
2023-01-22 21:08:59 +00:00
9ce117dd9f
chore: skip stealth test in github actions
2023-01-22 17:54:53 +00:00
a66cbcf6ee
feat(config)!: unsafe domain toggle ( #11588 )
2023-01-10 11:45:05 +00:00
d60d89ea87
feat(core): unify proxy; optimize request-wrapper ( #11472 )
...
* feat(core): unify proxy; optimize request-wrapper
Signed-off-by: Rongrong <i@rong.moe>
* Fix DeepScan issue
Signed-off-by: Rongrong <i@rong.moe>
* Simplify protocol existence judgment
Signed-off-by: Rongrong <i@rong.moe>
* chore: regenerate yarn.lock
Signed-off-by: Rongrong <i@rong.moe>
2022-12-23 01:13:01 +08:00
9264d7afbe
feat(core/utils/puppeteer-utils): Cookie utils ( #11471 )
...
Signed-off-by: Rongrong <i@rong.moe>
Signed-off-by: Rongrong <i@rong.moe>
2022-12-20 19:51:33 +08:00
1381377e91
fix(core/utils/puppeteer): proxy-chain error ( #11470 )
...
HTTPS/SOCKS proxy is not supported by proxy-chain and will cause an
error unconditionally. If an HTTPS/SOCKS proxy does not need
authentication, explicitly bypassing proxy-chain is a must. If it does
need, making a warning and continuing without proxy seems to be the best
choice.
Signed-off-by: Rongrong <i@rong.moe>
Signed-off-by: Rongrong <i@rong.moe>
2022-12-19 23:39:11 +08:00
058cec21b7
test(puppeteer): wait for bot test result ( #11322 )
...
Signed-off-by: Rongrong <i@rong.moe>
Signed-off-by: Rongrong <i@rong.moe>
2022-11-23 22:32:15 +00:00
6251a2dc02
fix(core): parse date utils ( #10789 )
...
* fix(core): parse date utils
* fix: regex
2022-09-14 21:59:32 +08:00
9329a44c80
feat(core): support proxy config for pupperteer ( #10714 )
...
* feat: support proxy config for pupperteer
* test: add puppeteer proxy detection
* fix: package.json
* test: fix regex
2022-09-07 00:07:04 +08:00
4cab5a5894
test: puppeteer wait longer ( #10599 )
2022-08-25 00:24:00 +08:00
bd80574f76
feat(route): add slowmist news ( #10044 )
...
* feat(route): add slowmist news
* feat(utils): remove useless ul for code section
* docs: add slowmist news route docs
* docs(quick-start): new router under /lib/v2
* test(utils): add test fixArticleContent for new wechat-mp code
* feat(router): update radar source of slowmist
2022-07-04 23:44:58 +08:00
1fcb5379df
fix(test): insufficiently robust puppeteer test ( #9974 )
...
Signed-off-by: Rongrong <i@rong.moe>
2022-06-17 21:55:33 +08:00
450e522167
feat(utils): decode cf protected email string ( #9900 )
...
* feat(utils): decode cf protected email string
* Update cf-email.js
2022-06-05 22:40:57 +08:00
13ec6be32b
fix(test): failing puppeteer without stealth test ( #9851 )
...
Signed-off-by: Rongrong <i@rong.moe>
2022-05-30 01:49:01 -03:00
0f31bfa8b9
fix(utils): parse relative date with meridiem ( #9775 )
...
* fix(utils): parse relative date with meridiem
* fix: use regex
2022-05-17 20:21:47 +08:00
2bcc6f6ad0
test: puppeteer wait until test finish
2022-05-04 20:43:15 +00:00
0b544e1395
feat(utils): puppeteer-extra-plugin-stealth ( #9676 )
2022-05-03 01:17:17 +08:00
0522c63d5f
fix(core/utils): invalid request header `Server` ( #9582 )
...
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-04-20 22:32:46 +08:00
bafb3534e1
feat(utils): random user agent ( #9449 )
...
* feat(utils): random ua
* chore: bump rand-user-agent to 1.0.58(no more deps)
2022-04-12 17:51:07 +08:00
74e1f88a32
feat(core)(utils/wechat-mp): normalize URL ( #9497 )
...
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-04-08 18:42:52 +08:00
a79cc20ec1
feat(utils): add utils for WeChat MP ( #9487 )
...
Motivation:
There are multiple routes that need to fetch articles from WeChat MP.
However, letting them fetch articles by themselves could potentially
lead to cache key collisions. Even if cache key collisions do not occur,
un-normalized URL could potentially lead to duplicated requests.
What's more, articles from WeChat MP have weird formats and need to be
fixed. Creating a universal function to do this work can create some
ease for new route contributors.
Note:
In order to make this PR atomic as much as possible, I did not touch
those broken routes. Once this PR is merged, I will try to fix them.
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-04-07 21:46:15 +08:00
5b35471e39
fix(core): offending RFC4287 ( #9441 )
...
* fix(core): offending RFC4287
should not leave `<updated>` blank when `<published>` is not blank
these two fields MUST conform to the "date-time" production in RFC3339
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
* test(common-utils): complete tests
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
* test(template): restrict expected value of `pubDate`
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-04-02 17:44:45 +08:00
bf834bdb7d
fix(utils): tests for parse-date ( #9428 )
2022-03-31 10:23:42 +08:00
d5648b37ee
fix(utils): parse relative dates with multiple time units ( #9365 )
...
* fix(utils): parse relative dates with multiple time units
* docs: remove warning
* fix: add more characters to match
* fix: rename to parse-date
2022-03-26 16:45:42 +08:00
0c6516c893
test: fix puppeteer error
2021-11-28 01:37:20 +00:00
d82847f541
style/chore(eslint): enforce new rules ( #8040 )
...
* style: prefer object shorthand syntax
* refactor: prefer Array#map over Array#forEach
* style: prefer arrow callback
* chore(eslint): update rules
* style: auto fix by eslint
2021-08-17 22:23:23 +08:00
6e3b58ed1d
refactor: avoid promise overhead ( #8028 )
2021-08-16 11:45:53 -07:00
1c9c4ccfc8
fix(core): make sure timeout error messages include URLs ( #7981 )
...
Before this fix, timeout messages are not quite useful
> error: Request undefined fail, retry attempt #1 : TimeoutError: Timeout awaiting 'request' for 5000ms
2021-08-12 01:27:07 -07:00
89e82d88fa
feat: got request timeout
2021-02-01 20:06:49 +08:00
68c15eaa5c
chore(deps): bump got from 11.1.4 to 11.3.0 ( #4922 )
...
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: DIYgod <diy.d.god@gmail.com>
2020-06-26 01:22:09 +08:00
6d7ab13150
test: longer closing puppeteer page time
2020-06-14 22:38:01 +08:00
910cbbc170
update docs & test cases
2020-05-06 23:12:43 +08:00
17e54c0ec8
fix: redundant moment dependence
2020-04-19 21:06:10 +08:00
0e7b14e718
chore(deps-dev): bump prettier from 1.19.1 to 2.0.2
2020-04-02 21:21:46 +08:00
48b27fe811
feat: auto add header Host
2020-02-15 21:32:16 +08:00
f6baa8d2cc
test: fix request-wrapper
2020-02-05 04:12:03 +08:00
77d55262ff
chore: bump got from 9.x to 10.x ( #3875 )
2020-02-04 01:09:14 +08:00
eec4fe626c
test: add date test case
2020-02-01 18:21:39 +08:00
da19f818a4
test: compatible with github actions
2019-11-25 21:59:24 +08:00
Tony
JimenezLi
DIYgod
dependabot[bot]
Rongrong
Atlas Quan
Rongrong
Rongrong
Ethan Shen
Sukka
Chih-Hsuan Yen
DIYgod
dependabot-preview[bot]
zengxs