Skip to content

Commit

Permalink
[WIP] Update Gatsbygram to make it work with current (unofficial) JSO…
Browse files Browse the repository at this point in the history
…N API. (gatsbyjs#3349)

* Update Gatsbygram to make it work with current (unofficial) JSON API.

* Revert back to previous data structure; add case study URL to README.
  • Loading branch information
Daniel Demmel authored and jastack committed Jan 24, 2018
1 parent b9791f5 commit 3b033cf
Show file tree
Hide file tree
Showing 32 changed files with 669 additions and 671 deletions.
4 changes: 4 additions & 0 deletions examples/gatsbygram/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,7 @@ https://gatsbygram.gatsbyjs.org/

Built with Gatsby 1.0 as a demo of Gatsby's new built-in image processing
capabilities.

To read more about this example, check out the [Gatsbygram Case Study][1]!

[1]: (https://www.gatsbyjs.org/blog/gatsbygram-case-study/)
Binary file modified examples/gatsbygram/data/images/4nm0fJs91a.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/6JCu-jM9yk.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/6TtTqhM981.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/7g4S2QM9_l.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/8HcJ7is90a.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/8rNAoWs9wR.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BBd11a1s9x3.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BCMVNmFM9yJ.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BEZprQ3s95w.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BFur3Jfs94V.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BGiWipNM96D.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BKwuIApgkKJ.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BLhhsTZlGwd.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BSEdRj_B8HP.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BSWUdcThRa-.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BShF_8qhtEv.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified examples/gatsbygram/data/images/BUHUK6ElhF1.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/gatsbygram/data/images/BW6uDlYlQCB.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/gatsbygram/data/images/BY5x5x2F6cO.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/gatsbygram/data/images/BY6B8z5lR1F.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/gatsbygram/data/images/Bam3G5XFOTK.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/gatsbygram/data/images/BdIpEzalAJx.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/gatsbygram/data/images/BdiU-TTFP4h.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed examples/gatsbygram/data/images/S6F7tDM99M.jpg
Binary file not shown.
Binary file removed examples/gatsbygram/data/images/SbHaprs9_K.jpg
Binary file not shown.
Binary file removed examples/gatsbygram/data/images/TEgqzWs98e.jpg
Diff not rendered.
Binary file removed examples/gatsbygram/data/images/TPga4IM9wF.jpg
Diff not rendered.
Binary file removed examples/gatsbygram/data/images/TWXrxBM98V.jpg
Diff not rendered.
1,288 changes: 644 additions & 644 deletions examples/gatsbygram/data/posts.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions examples/gatsbygram/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
"lint": "./node_modules/.bin/eslint --ext .js,.jsx --ignore-pattern public .",
"test": "echo \"Error: no test specified\" && exit 1",
"develop": "gatsby develop",
"dev": "gatsby develop",
"build": "gatsby build",
"deploy": "gatsby build --prefix-paths && gh-pages -d public"
}
Expand Down
47 changes: 20 additions & 27 deletions examples/gatsbygram/scrape.js
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
const fs = require("fs")
const request = require("request")
const mkdirp = require("mkdirp")
const ProgressBar = require("progress")
const { get } = require("lodash")
const download = require("./utils/download-file")
const fs = require(`fs`)
const request = require(`request`)
const mkdirp = require(`mkdirp`)
const ProgressBar = require(`progress`)
const { get } = require(`lodash`)
const download = require(`./utils/download-file`)

const username = process.argv[2]

Expand All @@ -14,7 +14,7 @@ You didn't supply an Instagram username!
Run this command like:
node scrape.js INSTAGRAM_USERNAME
`
`
)
process.exit()
}
Expand All @@ -38,36 +38,31 @@ let posts = []

// Write json
const saveJSON = _ =>
fs.writeFileSync(`./data/posts.json`, JSON.stringify(posts, "", 2))
fs.writeFileSync(`./data/posts.json`, JSON.stringify(posts, '', 2))

const getPosts = maxId => {
let url = `https://www.instagram.com/${username}/media`
if (maxId) url += `?max_id=${maxId}`
let url = `https://www.instagram.com/${username}/?__a=1`
if (maxId) url += `&max_id=${maxId}`

request(url, { encoding: `utf8` }, (err, res, body) => {
if (err) console.log(`error: ${err}`)
body = JSON.parse(body)
// Parse posts
let lastId
body.items
.filter(item => item.type === `image`)
body.user.media.nodes
.filter(item => item[`__typename`] === `GraphImage`)
.map(item => {
// Parse item to a simple object
return {
id: get(item, `id`),
code: get(item, `code`),
username: get(item, `user.username`),
avatar: get(item, `user.profile_picture`),
time: toISO8601(get(item, `created_time`)),
type: get(item, `type`),
time: toISO8601(get(item, `date`)),
type: get(item, `__typename`),
likes: get(item, `likes.count`),
comment: get(item, `comments.count`),
text: get(item, `caption.text`),
media: get(item, `images.standard_resolution.url`, ``).replace(
`/s640x640`,
``
),
text: get(item, `caption`),
media: get(item, `display_src`),
image: `images/${item.code}.jpg`,
username: get(body, `user.username`),
avatar: get(body, `user.profile_pic_url`),
}
})
.forEach(item => {
Expand All @@ -79,12 +74,10 @@ const getPosts = maxId => {

// Add item to posts
posts.push(item)

// Save lastId for next request
lastId = item.id
})

if (posts.length < 100 && get(body, `more_available`)) getPosts(lastId)
const lastId = get(body, `user.media.page_info.end_cursor`)
if (posts.length < 100 && lastId) getPosts(lastId)
else saveJSON()
})
}
Expand Down

0 comments on commit 3b033cf

Please sign in to comment.