Other modules: tracking progress #8

gadicc · 2021-01-30T17:21:38Z

Thanks to the awesome @roblav96 for all this beautiful work back over at pilwon/node-yahoo-finance#43 (in 2017 😅) Have added status [X]'s on each point to track progress, followed by any additional comments below.

Hey man, love what you're doing here. I've been reverse engineering Yahoo's API recently and thought I'd provide some of my findings.

[n/a] Crum Manager

sstrickx/yahoofinance-api
@sstrickx wrote a crum manager in Java.

As reported by @smeijer in pilwon/node-yahoo-finance#73 (thanks again!) and confirmed by me (and @MichaelDeBoey?), this is no longer required.

[X] Recommended Symbols

GET https://query1.finance.yahoo.com/v6/finance/recommendationsbysymbol/PLUG

Thanks, @pudgereyem, who added on 2021-02-09 in #28.

[ ] Daily Gainers

GET https://query2.finance.yahoo.com/v1/finance/screener/predefined/saved?formatted=false&lang=en-US&region=US&scrIds=day_gainers&count=5&corsDomain=finance.yahoo.com

[X] Search Autocomplete

GET https://autoc.finance.yahoo.com/autoc?query=PLUG&region=1&lang=en

[ ] Symbol Conversations

GET https://finance.yahoo.com/_finance_doubledown/api/resource/canvass.getMessageList;count=20;oauthConsumerKey=finance.oauth.client.canvass.prod.consumerKey;oauthConsumerSecret=finance.oauth.client.canvass.prod.consumerSecret;query=namespace = "yahoo_finance" and (tag = "PLUG");region=US;sortBy=createdAt;userActivity=true

[ ] Symbol News

GET http://feeds.finance.yahoo.com/rss/2.0/headline?s=PLUG&region=US&lang=en-US
Returns in RSS format, couldn't figure out how to get JSON.

[X] Multiple Symbol Quote

GET https://query2.finance.yahoo.com/v7/finance/quote?symbols=NVDA,PLUG,AMD

[ ] Multiple Symbol Historical Charts

GET https://query1.finance.yahoo.com/v7/finance/spark?symbols=PLUG,AMD&range=1d&formated=true
Only returns close data though =/

[X] Trending Symbols

GET https://query1.finance.yahoo.com/v1/finance/trending/US?lang=en-US&region=US&count=5&corsDomain=finance.yahoo.com

Thanks, @PythonCreator27, who added this on 2021-02-18 in #66.

[ ] Symbol Options

GET https://query1.finance.yahoo.com/v7/finance/options/[symbol]

Found by @PythonCreator27, more details below in #8 (comment).

Live Streamer

import * as Promise from 'bluebird'
import * as Nstream from 'stream'
import * as Nhttp from 'http'
import * as Nnet from 'net'
import * as Axios from 'axios'

const YH_IDK_KEY = {
	a00: 'Ask',
	a50: 'AskSize',
	b00: 'Bid',
	b60: 'BidSize',
	c10: 'Change',
	c63: 'ChangeRealtime',
	c64: 'DisputedChangeRealtimeAfterHours',
	c85: 'ChangeRealtimeAfterHours',
	c86: 'PercentChangeRealtimeAfterHours',
	g53: 'DayLow',
	h53: 'DayHigh',
	j10: 'MarketCap',
	l84: 'PriceRealtime',
	l86: 'PriceRealtimeAfterHours',
	p20: 'PercentChange',
	p43: 'PercentChangeRealtime',
	p44: 'PercentChangeRealtimeAfterHours',
	t53: 'DisputedTimestampForCommodities',
	t54: 'DisputedTimestampForStocks',
	v53: 'Volume',
	v00: 'Volume2',
	l10: 'LastSalePrice',
	t10: 'LastSaleTime',
	l90: 'EcnQuoteLastValue',
}
const YH_KEY_IDK = {
	Ask: 'a00',
	AskSize: 'a50',
	Bid: 'b00',
	BidSize: 'b60',
	Change: 'c10',
	ChangeRealtime: 'c63',
	DisputedChangeRealtimeAfterHours: 'c64',
	ChangeRealtimeAfterHours: 'c85',
	PercentChangeRealtimeAfterHours: 'p44',
	DayLow: 'g53',
	DayHigh: 'h53',
	MarketCap: 'j10',
	PriceRealtime: 'l84',
	PriceRealtimeAfterHours: 'l86',
	PercentChange: 'p20',
	PercentChangeRealtime: 'p43',
	DisputedTimestampForCommodities: 't53',
	DisputedTimestampForStocks: 't54',
	Volume: 'v53',
	Volume2: 'v00',
	LastSalePrice: 'l10',
	LastSaleTime: 't10',
	EcnQuoteLastValue: 'l90',
}

let total = 0
let stream = null as Nhttp.IncomingMessage
function startYahooStream() {
	return Promise.resolve().then(function() {
		let keys = [
			YH_KEY_IDK.Ask,
			YH_KEY_IDK.AskSize,
			YH_KEY_IDK.Bid,
			YH_KEY_IDK.BidSize,
			YH_KEY_IDK.Volume,
			YH_KEY_IDK.Volume2,
			YH_KEY_IDK.PriceRealtime,
			YH_KEY_IDK.PriceRealtimeAfterHours,
			YH_KEY_IDK.LastSalePrice,
			YH_KEY_IDK.EcnQuoteLastValue,
			YH_KEY_IDK.LastSaleTime,
			YH_KEY_IDK.DisputedTimestampForStocks,
		]
		return Axios.default.get('https://streamerapi.finance.yahoo.com/streamer/1.0', {
			params: {
				s: 'NVDA,PLUG,AMD',
				k: keys.join(','),
				r: 0,
				callback: 'parent.yfs_u1f',
				mktmcb: 'parent.yfs_mktmcb',
				gencallback: 'parent.yfs_gencb',
				mu: 1,
				lang: 'en-US',
				region: 'US',
				localize: 0,
			},
			responseType: 'stream',
		}).then<any>(function({ data }) {
			return new Promise<string>(function(resolve, reject) {
				stream = data as Nhttp.IncomingMessage

				stream.on('data', function(chunk) {
					try {
						let input = chunk.toString()
						let jsons = [] as Array<{ [key: string]: any }>

						let split = input.split('yfs_u1f(')
						split.shift()
						total = total + split.length
						split.forEach(function(v) {
							let i = v.indexOf(');}')
							let parsed = v.substring(0, i).replace(/(['"])?([a-zA-Z0-9_]+)(['"])?:/g, '"$2": ')
							if (!parsed) return;
							jsons.push(JSON.parse(parsed))
						})

						let sendi = {}
						jsons.forEach(function(s) {
							let symbol = Object.keys(s)[0]
							if (!sendi[symbol]) sendi[symbol] = {};
							let v = s[symbol]
							Object.keys(v).forEach(function(k) {
								let n = parseFloat(v[k])
								if (n == 0) return;
								let key = YH_IDK_KEY[k] as string
								sendi[symbol][key] = n
							})
						})
						console.log('sendi >', JSON.stringify(sendi, null, 4))

					} catch (error) {
						console.error('quotes.startYahooStream > catch > error', error)
					}
				})

				stream.on('end', resolve)
				stream.on('error', reject)
			})
		})

	}).catch(function(error) {
		console.error('quotes.startYahooStream > error', error)
		return Promise.resolve()

	}).finally(function() {
		console.warn('quotes.stream > FINALLY')
		if (stream) {
			stream.removeAllListeners()
			stream.destroy()
			stream = null
		}
		return startYahooStream()
	})
}
startYahooStream()

console.log('sendi >', JSON.stringify(sendi, null, 4))

sendi >
{
    NVDA: {
        LastSalePrice: 149.6,
        Volume2: 92,
        Ask: 149.5,
        AskSize: 1800,
        Bid: 148.2,
        BidSize: 100,
        PriceRealtime: 149.6
    }
}

Hope you find this helpful!

The text was updated successfully, but these errors were encountered:

gadicc · 2021-02-08T08:03:26Z

Added quote module in e0f8d35...1cbc17a, released in v1.6.0.

pudgereyem · 2021-02-08T22:04:46Z

Created a PR that adds a new module for recommendationsBySymbol here; #28

pudgereyem · 2021-02-10T09:43:13Z

New module recommendationsBySymbol was added in with #28.

@gadicc, would you mind updating the Issue to reflect this since I can't do it myself? 🙏

gadicc · 2021-02-10T14:22:04Z

Thanks, @pudgereyem! Have updated the checklist. Thanks again for all your awesome work on this.

advaiyalad · 2021-03-19T02:05:52Z

@gadicc Found another endpoint: https://query1.finance.yahoo.com/v7/finance/options/[symbol]. Not sure if this was found already.

Query Params

Param Name	Param Type	Required
formatted	Boolean	false
crumb	String	false
lang	String	false
region	String	false

corsDomain omitted as it does nothing.

Another finding: query1, query2, and iquery are the same. We can fetch from query1.finance.yahoo.com OR query2.finance.yahoo.com OR iquery.finance.yahoo.com for the same data. Probably want to unify what we use in our code, or allow users to choose from each domain.

Hope this helps!

gadicc · 2021-03-19T06:43:55Z

Hey, that's awesome, thanks @PythonCreator27! Have added it to the list in the first post.

And for noticing about the alternative host names, we should indeed accommodate that.

roblav96 · 2021-03-27T00:38:08Z

@gadicc Appreciate the creds friend =]

Take a peek at https://github.com/roblav96/robinhood.tools it has some additional sources of financial data n more.

roblav96 · 2021-03-27T00:44:55Z

More specifically:

For example:

curl 'https://quoteapi.webull.com/api/quote/tickerRealTimes/full?tickerIds=925334567,925353501,913255891'

gadicc · 2021-03-27T06:40:38Z

Hey @roblav96, great to see you're still involved with all this stuff, and to see you on this new repo!

Thanks for sharing the above. I recall on the original repo too people were looking for additional sources, and I guess we should add this to the README or an FAQ or something, so thank you!

What is your rough breakdown of consumption from Yahoo vs Webull now? Just out of curiosity.

roblav96 · 2021-03-30T12:33:04Z

@gadicc Webull provides the best scrapable market data, but they make you work for it. lol Start by exploring the network traffic of https://app.webull.com/ using your web browser's network debug tools.

advaiyalad · 2021-04-09T19:33:04Z

Another note: recommendationsBySymbol is using an older version of the endpoint. The newer version is: https://query1.finance.yahoo.com/v7/finance/recommendationsbysymbol/PLUG. Yahoo's website does seem to use the older version, though. Changing the endpoint, though, would be a definite breaking change. @gadicc Do you think that it is worth it, or should we just create a brand new module for the newer version of the endpoint? The newer endpoint seems to just return a Quote[] instead of its own response format.

gadicc · 2021-04-10T05:56:32Z

Well found, @PythonCreator27. I would just leave it for now, from what you've said (and thanks for checking this all out!), there doesn't seem to be a compelling reason to upgrade. If something happens to the v6 one, maybe we could switch to v7 but return the results in the same format if that's the only difference (and our validation checks will let us know if we missed anything).

It is something to think about though for all modules. Maybe in future we should name the modules with Yahoo's API version, e.g. recommendationsBySymbolV7(). That would make it easier to cover these kinds of cases without needing to publish a new major version of the library just for one end point. Is a bit verbose though.

In short, I suggest to leave for now and see what other cases come up in the future and then make a call. Thanks for bringing attention to this.

gadicc · 2021-04-10T06:10:51Z

@gadicc Webull provides the best scrapable market data, but they make you work for it. lol Start by exploring the network traffic of https://app.webull.com/ using your web browser's network debug tools.

Thanks, @roblav96! I've made a start of referencing this (a long with some of the older suggestsions from node-yahoo-finance (v1) issues) on the Wiki: https://github.com/gadicc/node-yahoo-finance2/wiki/Alternative-Sources

I invite anyone to help improve this page with anything useful.

advaiyalad · 2021-04-16T02:36:12Z

Another finding (endpoint): https://query2.finance.yahoo.com/ws/insights/v2/finance/insights?lang=en-US&region=US&symbol=AMD&getAllResearchReports=true&reportsCount=2

Insights - seems interesting. I was looking for answers to #138, and found this along the way... Will send a PR soon to add this.

pudgereyem mentioned this issue Feb 5, 2021

Should result validation be optional #16

Closed

advaiyalad mentioned this issue Feb 17, 2021

feat(modules): add trendingSymbols module #66

Merged

gadicc mentioned this issue Mar 19, 2021

Use both query1 and query2. #95

Closed

advaiyalad mentioned this issue Apr 7, 2021

GitHub Issue (and PR) Templates, Labels, GH Discussions #116

Closed

6 tasks

z3nful mentioned this issue Sep 6, 2023

feat(module): add dailyGainers #666

Merged

6 tasks

StephanAkkerman mentioned this issue Aug 7, 2024

Use custom yahoo finance for async compatability StephanAkkerman/fintwit-bot#614

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Other modules: tracking progress #8

Other modules: tracking progress #8

gadicc commented Jan 30, 2021 •

edited

Loading

[n/a] Crum Manager

[X] Recommended Symbols

[ ] Daily Gainers

[X] Search Autocomplete

[ ] Symbol Conversations

[ ] Symbol News

[X] Multiple Symbol Quote

[ ] Multiple Symbol Historical Charts

[X] Trending Symbols

[ ] Symbol Options

Live Streamer

console.log('sendi >', JSON.stringify(sendi, null, 4))

gadicc commented Feb 8, 2021 •

edited

Loading

pudgereyem commented Feb 8, 2021

pudgereyem commented Feb 10, 2021 •

edited

Loading

gadicc commented Feb 10, 2021 •

edited

Loading

advaiyalad commented Mar 19, 2021 •

edited

Loading

gadicc commented Mar 19, 2021

roblav96 commented Mar 27, 2021

roblav96 commented Mar 27, 2021

gadicc commented Mar 27, 2021

roblav96 commented Mar 30, 2021

advaiyalad commented Apr 9, 2021

gadicc commented Apr 10, 2021

gadicc commented Apr 10, 2021

advaiyalad commented Apr 16, 2021

Other modules: tracking progress #8

Other modules: tracking progress #8

Comments

gadicc commented Jan 30, 2021 • edited Loading

[n/a] Crum Manager

[X] Recommended Symbols

[ ] Daily Gainers

[X] Search Autocomplete

[ ] Symbol Conversations

[ ] Symbol News

[X] Multiple Symbol Quote

[ ] Multiple Symbol Historical Charts

[X] Trending Symbols

[ ] Symbol Options

Live Streamer

console.log('sendi >', JSON.stringify(sendi, null, 4))

gadicc commented Feb 8, 2021 • edited Loading

pudgereyem commented Feb 8, 2021

pudgereyem commented Feb 10, 2021 • edited Loading

gadicc commented Feb 10, 2021 • edited Loading

advaiyalad commented Mar 19, 2021 • edited Loading

Query Params

gadicc commented Mar 19, 2021

roblav96 commented Mar 27, 2021

roblav96 commented Mar 27, 2021

gadicc commented Mar 27, 2021

roblav96 commented Mar 30, 2021

advaiyalad commented Apr 9, 2021

gadicc commented Apr 10, 2021

gadicc commented Apr 10, 2021

advaiyalad commented Apr 16, 2021

gadicc commented Jan 30, 2021 •

edited

Loading

gadicc commented Feb 8, 2021 •

edited

Loading

pudgereyem commented Feb 10, 2021 •

edited

Loading

gadicc commented Feb 10, 2021 •

edited

Loading

advaiyalad commented Mar 19, 2021 •

edited

Loading