f3e069d399
fix(utils): 支持微信公众号单图片文章抓取; 增加输出阅读原文链接 ( #9557 )
...
* fix(utils): 支持微信公众号单图片文章抓取
* fix(utils): 支持输出微信公众号转载文章阅读原文链接
2022-04-19 01:29:45 +08:00
958be6266e
feat(route): 山东大学（威海）新闻网 ( #9537 )
...
* feat(sduwh): add extractors.
* feat(route): add route for 山东大学（威海）新闻网
* docs: for route sduwh/news
* docs: for route sduwh/news
(cherry picked from commit 831830167a39a658520dd9a46b375fdbbcb5d611)
* feat(radar): for route 山东大学（威海）新闻网
* refactor: change `got.get` to `got`.
* refactor: prefer `parseDate()` to `new Date()`
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
* fix: incomplete URL substring sanitization.
Make CodeQL happy.
* fix(radar): fix target field.
* fix: change route /sduwh to /sdu/wh
* fix: remove superfluous slash character in url.
* feat: look for exact date first.
* feat: extract exact date from news extractor.
* feat: extract exact date from view extractor.
* feat: extractor for www.sdrj.sdu.edu.cn
* refactor: semantic separation of sduwh with sdu
* feat(radar): more accurate name
* docs: update documentation
* refactor: migrate to v2
* refactor: fix deprecated url.resolve
* fix: update docs url
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
* fix: sdu not working routes
* fix: accurate `ctx.state.data.url`
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
* fix: better error handling for extractors.
* fix: timezone
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
* fix: better error handling.
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
2022-04-17 00:01:39 +08:00
eb467afae1
fix(utils): 支持将微信公众号转载文章的正文抓取回来 ( #9534 )
...
* feat: 修正日期时间匹配规则、移除一些不必要评论元素
* fix(route)(fortunechina): 修正财富中文网1、双语文章中文内容重复问题；2、移除 kol 大头像
* fix(route)(wechat): 支持将微信公众号转载文章的正文抓取回来
2022-04-15 19:23:09 +08:00
bafb3534e1
feat(utils): random user agent ( #9449 )
...
* feat(utils): random ua
* chore: bump rand-user-agent to 1.0.58(no more deps)
2022-04-12 17:51:07 +08:00
a25fd4b67f
fix(utils): wrong date with same weekday in parse-date ( #9506 )
2022-04-10 23:40:35 +08:00
ef94bcde8e
feat(route): add two blogs: Stratechery & Miris Whispers ( #9496 )
...
* Add two blogs
* fix(utils): add parseDate timezone common-config
* Update lib/v2/stratechery/index.js
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
* Update docs/blog.md
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
* Update docs/en/blog.md
Co-authored-by: Tony <TonyRL@users.noreply.github.com>
* Update lib/v2/miris/blog.js
2022-04-08 23:48:28 +08:00
74e1f88a32
feat(core)(utils/wechat-mp): normalize URL ( #9497 )
...
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-04-08 18:42:52 +08:00
a79cc20ec1
feat(utils): add utils for WeChat MP ( #9487 )
...
Motivation:
There are multiple routes that need to fetch articles from WeChat MP.
However, letting them fetch articles by themselves could potentially
lead to cache key collisions. Even if cache key collisions do not occur,
un-normalized URL could potentially lead to duplicated requests.
What's more, articles from WeChat MP have weird formats and need to be
fixed. Creating a universal function to do this work can create some
ease for new route contributors.
Note:
In order to make this PR atomic as much as possible, I did not touch
those broken routes. Once this PR is merged, I will try to fix them.
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-04-07 21:46:15 +08:00
5b35471e39
fix(core): offending RFC4287 ( #9441 )
...
* fix(core): offending RFC4287
should not leave `<updated>` blank when `<published>` is not blank
these two fields MUST conform to the "date-time" production in RFC3339
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
* test(common-utils): complete tests
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
* test(template): restrict expected value of `pubDate`
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-04-02 17:44:45 +08:00
95db3b4e99
fix(core): torrent searching error ( #9407 )
...
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-03-29 21:12:16 +08:00
cd151f45ab
fix(utils): parseDate custom format not working
2022-03-28 22:25:40 +08:00
a3396029ec
fix(utils): wrong last weekday for relative date ( #9397 )
2022-03-27 23:39:44 +08:00
e42bdf3d97
fix(utils): typo in parse-date ( #9391 )
...
* fix(utils): typo in parse-date
* fix: add `后日`
2022-03-26 19:44:27 +08:00
d5648b37ee
fix(utils): parse relative dates with multiple time units ( #9365 )
...
* fix(utils): parse relative dates with multiple time units
* docs: remove warning
* fix: add more characters to match
* fix: rename to parse-date
2022-03-26 16:45:42 +08:00
c48ca6bd5b
fix(core): invalid feed fields ( #9286 )
...
Signed-off-by: Rongrong <15956627+Rongronggg9@users.noreply.github.com>
2022-03-22 02:13:15 +08:00
2c2bf4ed09
fix: typo in puppeteer options
2022-03-06 23:42:32 +08:00
35c9834049
fix: puppeteer userDataDir
2021-11-28 01:21:42 +00:00
0792f7ba25
feat(core): first attempt to init script standard ( #8224 )
...
- lazy load
- rate limit per path
- init .debug.json support
- docs
- maintainer
- radar
2021-09-22 05:41:00 -07:00
d77a039f05
style(eslint): add no-implicit-coercion rule ( #8175 )
...
* refactor: add no-implicit-coercion rule for ESLint
* fix: errors from deepscan
* fix: errors from deepscan
* fix: errors from deepscan
* fix: errors from deepscan
* fix: errors from deepscan
* Update docs/en/joinus/quick-start.md
Co-authored-by: Sukka <isukkaw@gmail.com>
* Update docs/joinus/quick-start.md
Co-authored-by: Sukka <isukkaw@gmail.com>
* Update lib/routes/av01/tag.js
Co-authored-by: Sukka <isukkaw@gmail.com>
* Update lib/routes/gov/taiwan/mnd.js
Co-authored-by: Sukka <isukkaw@gmail.com>
* Update lib/routes/ps/product.js
Co-authored-by: Sukka <isukkaw@gmail.com>
* refactor: minify html string
Co-authored-by: Sukka <isukkaw@gmail.com>
2021-09-15 21:22:11 +08:00
31720bbb1b
perf: lazy require dependencies ( #8025 )
2021-08-20 14:05:57 +08:00
d82847f541
style/chore(eslint): enforce new rules ( #8040 )
...
* style: prefer object shorthand syntax
* refactor: prefer Array#map over Array#forEach
* style: prefer arrow callback
* chore(eslint): update rules
* style: auto fix by eslint
2021-08-17 22:23:23 +08:00
6e3b58ed1d
refactor: avoid promise overhead ( #8028 )
2021-08-16 11:45:53 -07:00
1c9c4ccfc8
fix(core): make sure timeout error messages include URLs ( #7981 )
...
Before this fix, timeout messages are not quite useful
> error: Request undefined fail, retry attempt #1 : TimeoutError: Timeout awaiting 'request' for 5000ms
2021-08-12 01:27:07 -07:00
65e74a1c5e
chore(utils): `parse-date` supports relative time & fix routes ( #7530 )
2021-05-14 23:08:33 -04:00
e1b3b5d877
style: auto format
2021-05-08 21:49:05 +00:00
10f5bb7bce
refactor: timezone conversion in `lib/utils/date.js` ( #7438 )
2021-05-08 17:45:37 -04:00
89e82d88fa
feat: got request timeout
2021-02-01 20:06:49 +08:00
c5e3a27f44
feat: auto add headers.host
2020-12-17 18:54:29 +08:00
d4bdf8c7e8
feat: add 豆瓣用户想看 ( #6285 )
2020-12-04 15:44:33 +00:00
3442ca9196
feat: more readable twitter tweet ( #6051 )
2020-10-30 09:09:10 +00:00
ec49562269
Revert #5271
...
Incomplete PR. #5261
2020-07-29 17:25:41 +01:00
eddba23099
feat: javbus routes: `director` `label` `studio` ( #5271 )
...
Co-authored-by: zrenca <42361841+zrenca@users.noreply.github.com>
2020-07-28 15:59:30 +01:00
90030ec017
fix anitama date ( #5255 )
2020-07-28 05:54:43 +01:00
68c15eaa5c
chore(deps): bump got from 11.1.4 to 11.3.0 ( #4922 )
...
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: DIYgod <diy.d.god@gmail.com>
2020-06-26 01:22:09 +08:00
ae529572d7
feat: longer closing puppeteer page time
2020-06-14 22:28:43 +08:00
f7a1380dac
fix: host in headers after redirecting request ( #4717 )
2020-05-11 18:35:08 +08:00
8f371e6703
Merge branch 'proxy' of git://github.com/zengxs/RSSHub into zengxs-proxy
2020-05-07 00:42:56 +08:00
58e930b4cc
support proxy-uri
2020-05-06 22:43:09 +08:00
725a733b2a
fix: ingore undefined got options ( #4651 )
2020-05-06 19:47:51 +08:00
fedfc741d5
feat: add config noLogfiles, set noLogFiles for now
2020-04-28 18:21:36 +08:00
7902c547cf
feat: depoly on now serverless function ( #4554 )
2020-04-28 17:28:57 +08:00
17e54c0ec8
fix: redundant moment dependence
2020-04-19 21:06:10 +08:00
0e7b14e718
chore(deps-dev): bump prettier from 1.19.1 to 2.0.2
2020-04-02 21:21:46 +08:00
0b63142add
test: fix undefined url
2020-02-15 21:46:10 +08:00
48b27fe811
feat: auto add header Host
2020-02-15 21:32:16 +08:00
77d55262ff
chore: bump got from 9.x to 10.x ( #3875 )
2020-02-04 01:09:14 +08:00
fbabb3679c
feat: add route for netease god ( #3801 )
2020-02-01 18:11:02 +08:00
252f314a19
feat: 增加Discuz通用路由 ( #3721 )
2020-01-16 14:20:10 +08:00
7a2dc47bf7
feat(core): auto referer
2019-10-22 17:25:32 +08:00
2204388bbd
fea: webtoons ( close #3164 ) ( #3178 )
2019-10-09 11:02:11 +08:00
4ae28e48b5
feat: disable console log for pkg
2019-09-18 17:57:05 +08:00
4445443946
feat: use dynamical config
2019-09-18 17:45:21 +08:00
4d6257af50
feat: replace baseUrl to prefixUrl when got's next version ( #2996 )
...
* enhance: replace baseUrl to prefixUrl when got's next version
* test: baseUrl
2019-09-05 12:01:44 +08:00
70c9546871
fix: puppeteer close error
2019-09-03 23:04:42 +08:00
a34b9ae542
feat: remove addNoReferrer
2019-08-28 14:33:05 +08:00
29f588e6a0
feat: handle description in parameter middleware
2019-08-28 12:06:50 +08:00
6f2ee73073
feat: move encodeURIComponent to routes
2019-08-18 18:03:41 +08:00
eba8f46875
feat: header x-app -> server
2019-08-15 23:14:48 +08:00
d6fccdeeac
feat: queshu title add book name ( #2862 )
2019-08-15 22:19:00 +08:00
17b4946c00
feat: add queshu ( #2850 )
...
* feat: add queshu
* feat: add queshu
2019-08-14 17:34:21 +08:00
3ce63c72b7
fix: try to solve #1888 ( #2829 )
2019-08-11 23:12:54 +08:00
0aaf1e3f6c
fix: object got query, close #2824
2019-08-11 01:33:33 +08:00
5d8cf35ef5
Merge branch 'request-wrapper'
2019-08-11 00:37:34 +08:00
e5d34eda6b
test: request-wrapper
2019-08-11 00:31:20 +08:00
054d159e8c
fix: Request path contains unescaped characters error ( #2784 )
2019-08-08 22:49:32 +08:00
6f3a7836f2
feat: request wrapper
2019-08-06 13:00:22 +08:00
08878e1d52
fix: set proxy for http module, close #2647 #2094
2019-08-03 20:38:37 +08:00
cd10dfcc93
refactor: format
2019-08-02 17:55:00 +08:00
914034760d
feat: 新增自动提取全文输出 ( #2738 )
2019-08-02 17:22:58 +08:00
1d071836a9
fix: various optimizations ( #2581 )
...
* chore: string quotations
* fix: various optimizations
remove redundant checks
2019-07-08 11:52:47 +08:00
d09affa703
feat: add addNoReferrer in common-utils ( #2571 )
...
* fix: no-referrer tag
* chore: merge qdaily routes
* feat: add addNoReferrer in common-utils
This adds no-referrer attribute to images.
Parameters:
- add no-referrer attribute to images
- $: cheerio selector
- source: source selector, string
- target: target attribute name, typically for lazyload imgs, string
- srcPrefix: prefix for src, string
- removeAttr: attributes to remove, array: ['attrA','attrB']
* test: add test case
* doc: update
2019-07-08 11:45:02 +08:00
59bb15e761
fix: got encoding
2019-06-15 10:24:06 +08:00
56e74742fc
test: got axios options
2019-06-14 17:51:03 +08:00
749ede3af8
test: puppeteer auto close
2019-06-13 18:37:55 +08:00
f8f427a414
test: date cases
2019-06-13 16:21:05 +08:00
4531de80f9
feat: add common-utils ( #2374 )
2019-06-12 11:09:27 +08:00
ea508b8bc2
fix: post type
2019-06-04 18:28:06 +08:00
907d7498ca
fix: compatible with axios data and params
2019-06-04 15:10:27 +08:00
3efdb57be0
fix: got option.body
2019-06-04 11:59:36 +08:00
feab46cb15
fix: got.all
2019-06-04 11:54:20 +08:00
2bd3b39fcd
fix: always parse response body
2019-06-04 11:13:54 +08:00
a56c0b5a39
refactor: rename axios to got
2019-06-03 18:03:05 +08:00
c4bf4fcdb4
fix: agent option
2019-06-03 17:44:22 +08:00
163a589e20
Merge branch 'master' into feature/got
2019-06-03 17:06:48 +08:00
633154c778
refactor: 重构 rss-parser 作为工具类 ( #2280 )
2019-06-02 12:45:52 +08:00
e2badbfa98
chore: module alias for all routes
2019-05-15 15:27:15 +08:00
4e5f50fe56
feat: OSChina 新闻抓取的时候解析时间字符串 ( #2062 )
2019-05-07 12:08:51 +08:00
df026d7198
support http and https proxy
2019-04-18 14:27:07 +08:00
07b2771b63
fix buffer encoding
2019-04-18 00:20:39 +08:00
57a06b190f
Merge branch 'use-got-fix-proxy' of https://github.com/fengkx/RSSHub into fengkx-use-got-fix-proxy
2019-04-17 23:05:13 +08:00
9ff646c066
fix douyin
2019-04-11 14:35:16 +08:00
ba57ded1cd
解决一分钟之内发布的微博，pubDate为'刚刚'的bug ( #1874 )
...
解决一分钟之内发布的微博，pubDate为'刚刚'的bug
close #1870
2019-04-08 11:28:27 +08:00
ee137c4007
增加艾瑞产业研究报告
2019-03-25 13:10:48 +08:00
f4a958d93a
format
2019-03-23 00:41:40 +08:00
f8990b7cb7
The cherrio config route ( #1699 )
...
Use js config too build rss
close #349 close #1035
2019-03-11 10:47:09 +08:00
ca0b9bca33
format code
2019-03-10 09:37:15 +08:00
1c0bf5697f
test
2019-03-10 00:19:54 +08:00
a1259483cd
replace axios using got
2019-02-28 21:51:24 +08:00
19fa9093af
add segmentfault频道 ( #1543 )
2019-02-19 11:50:42 +08:00
58f0333cca
test: date
2019-01-23 11:55:50 +08:00
任平生
Levi Zim
Tony
Ethan Shen
Chenxing Luo
Rongrong
DIYgod
NeverBehave
Daniel Li (李丹阳)
Sukka
Chih-Hsuan Yen
Queensferry
GitHub Action
DIYgod
Herb Brewer
Shun Zi
Henry
sabuaka18
hoilc
dependabot-preview[bot]
Mitt
zengxs
fengkx
Luyu Huang
junfengP
MachX
Cloud
sanmmmm
Xiang Li
zengxs
footair
Marcus
SettingDust
Chenyang Shi