How would I go about downloading the contents of a URL in Node when using the Express framework? Basically, I need to complete the Facebook authentication flow, but I can't do this without GETing their OAuth Token URL.
Normally, in PHP, I'd use Curl, but what is the Node equivalent?
asked Oct 14, 2011 at 19:27
var options = { host: 'www.google.com', port: 80, path: '/index.html' }; http.get(options, function(res) { console.log("Got response: " + res.statusCode); }).on('error', function(e) { console.log("Got error: " + e.message); });
//nodejs.org/docs/v0.4.11/api/http.html#http.get
answered Oct 14, 2011 at 21:36
chovychovy
67.8k48 gold badges204 silver badges260 bronze badges
4
The problem that you will front is: some webpage loads its contents using JavaScript. Thus, you needs a package, like After-Load which simulates browser's behavior, then gives you the HTML content of that URL .
var afterLoad = require('after-load'); afterLoad('//google.com', function(html){ console.log(html); });
Sergio
28.2k10 gold badges83 silver badges130 bronze badges
answered Jul 27, 2016 at 5:12
Abdennour TOUMIAbdennour TOUMI
79.8k36 gold badges233 silver badges241 bronze badges
1
Using http way requires way more lines of code for just a simple html page .
var request = require("request"); request({uri: "//www.sitepoint.com"}, function(error, response, body) { console.log(body); }); });Here's an efficient way : Use request
Here is the doc for request : //github.com/request/request
2nd Method using fetch with promises :
fetch('//sitepoint.com') .then(resp=> resp.text()).then(body => console.log(body)) ;answered Jun 23, 2018 at 10:50
Natesh bhatNatesh bhat
11k10 gold badges74 silver badges115 bronze badges
Using http module:
const http = require('http'); http.get('//localhost/', (res) => { let rawHtml = ''; res.on('data', (chunk) => { rawHtml += chunk; }); res.on('end', () => { try { console.log(rawHtml); } catch (e) { console.error(e.message); } }); });rawHtml - complete html of the page.
I just simplified example from official docs.
answered Jul 2, 2021 at 19:20
ScofieldScofield
3,6551 gold badge23 silver badges29 bronze badges
using Axios is much simpler
const axios = require("axios").default const response = axios.get("//google.com") console.log(response.data)or
const axios = require("axios").default const response = axios.get("//google.com").then((response)=>{ console.log(response.data) })for full docs, you can head over Axios Github
answered Mar 28 at 8:51
1
Get the HTML from any website, using prerendering when is necessary.
Features
- Get HTML markup from any website (client side apps as well).
- Prerendering detection based on domains list.
- Speed up process blocking ads trackers.
- Encoding body response properly.
Headless technology like puppeteer brings us to get the HTML markup from any website, even when the target URL is client side app and we need to wait until dom events fire for getting the real markup.
Generally this approach better than a simple GET request from the target URL, but because you need to wait for dom events, prerendering could be slow and in some scenario unnecessary (sites that use server side rendering could be resolved with a simple GET).
html-get bring the best of both worlds, doing the following algorithm:
- Determinate if the target URL actually needs prerendering (internally it has a list of popular site domains that don't need it).
- If it needs prerendering, perform the action using Headless technology, blocking ads trackers requests for speed up the process, trying to resolve the main request in the minimum amount of time.
- If it does not need prerendering or prerendering fails for any reason (for example, timeout), the request will be resolved doing a GET request.
Install
$ npm install browserless puppeteer html-get --save
Usage
const createBrowserless = require('browserless') const getHTML = require('html-get') // Spawn Chromium process once const browserlessFactory = createBrowserless() // Kill the process when Node.js exit process.on('exit', () => { console.log('closing resources!') browserlessFactory.close() }) const getContent = async url => { // create a browser context inside Chromium process const browserContext = browserlessFactory.createContext() const getBrowserless = () => browserContext const result = await getHTML(url, { getBrowserless }) // close the browser context after it's used await getBrowserless((browser) => browser.destroyContext()) return result } getContent('//example.com') .then(content => { console.log(content) process.exit() }) .catch(error => { console.error(error) process.exit(1) })
Command Line
$ npx html-get //example.comAPI
getHTML(url, [options])
url
Required
Type: string
The target URL for getting the HTML markup.
getBrowserlessRequired
Type: function
A function that should return a browserless instance to be used for interact with puppeteer:
options
prerenderType: boolean|string
Default: 'auto'
Enable or disable prerendering as mechanism for getting the HTML markup explicitly.
The value auto means that that internally use a list of websites that don't need to use prerendering by default. This list is used for speedup the process, using fetch mode for these websites.
See getMode parameter for know more.
encodingType: string
Default: 'utf-8'
Encoding the HTML markup properly from the body response.
It determines the encode to use A Node.js library for converting HTML documents of arbitrary encoding into a target encoding (utf8, utf16, etc).
headersType: object
Request headers that will be passed to fetch/prerender process.
getModeType: function
A function evaluation that will be invoked to determinate the resolutive mode for getting the HTML markup from the target URL.
The default getMode is:
const getMode = (url, { prerender }) => { if (prerender === false) return 'fetch' if (prerender !== 'auto') return 'prerender' return autoDomains.includes(getDomain(url)) ? 'fetch' : 'prerender' }
gotOptionsType: object
Under mode=fetch, pass configuration object to got.
puppeteerOptsType: object
Under non mode=fetch, pass configuration object to puppeteer.
rewriteUrlsType: boolean
Default: false
When is true, it will be rewritten CSS/HTML relatives URLs present in the HTML markup into absolutes.
License
html-get © Microlink, released under the MIT License.
Authored and maintained by
Kiko Beats with help from contributors.
microlink.io · GitHub microlinkhq · Twitter @microlinkhq