Category: gist

There are a lot of songs on YouTube. To make a copy of just the audio of a video on your computer, follow these steps:

1. Install youtube-dl. If you’re using a mac and use Homebrew, you can simply type brew install youtube-dl
2. Then from the command line, type youtube-dl --extract-audio --audio-format mp3 <video URL>

I was using the freshbooks npm module, but started having issues with its dependencies as it was last updated 3 years ago.

It turns out even though the payloads are in XML, it’s not too bad calling their classic API directly once you know how. Of course, then you need to process an XML response. But instead of having to use an XML library, since I know what specific fragment of data I want, I can use regular expressions to extract it.

So here is a code sample on how to call their API directly

Let’s say you have a lot of query parameters you need to add to an Ajax request. For code maintainability, it’s much cleaner to create a large JavaScript object, use string interpolation for the URL, and then adding a function that converts the JavaScript object to a query string.

This is what it might look like…

YouTube automatically creates subtitles for a lot of videos that are uploaded to YouTube.

So instead of paying to have your videos transcribed, you can upload your video to YouTube, and programatically download the subtitles.

But it took me a while to figure out how to get those subtitles programatically.

I started with youtube-dl, but I could only get the WebVTT format which is cumbersome to post-process. I posted an issue in their GitHub repo.

I finally found a little npm library someone had built that made this task fairly straight forward. Here is a code sample that will take a YouTube URL as a command line argument, and then log the subtitles as plain text.

If you’re creating an app where you need to access Google services from a server, you’ll need to store the refresh token from the Google Oauth process. The access tokens periodically expire, so you need the refresh token to manually get a new access token when you need it.

So if you’re using the Passport auth middleware for Node.js you would have something like this to make sure you get the refresh token. Store this refresh token in a database.

Then when you need an access token to access gmail in this case, you could get a new access token like this

— based on these Google Identity Platform docs

If you have a front-end that needs to accept a CSV file, upload it to a server using jQuery, and then have that Node.js server process that CSV file, then this a way to do it.

First up, the HTML

and then the jQuery that takes the selected CSV file, and uploads it to a server

and finally the Node.js file that will accept and process it using Express (with the multer middleware), and then iterate through the CSV file.

As it turns out, the Node.js CSV libraries I found on npm were either not working or hadn’t been updated in a long time. To process a CSV file with standard JS using functions like split and replace made this relatively easy to do on my own.

Any questions let me know!

 

I had an issue where I needed to scrape some data from a dynamically generated webpage. Open this page in a browser and you’ll see what I mean.

I tried a lot of options, and the solution that ended up working for me was to use Headless Chrome. This basically allows you to launch Chrome and use it as a tool from within your code. It’s recommended to install Chrome Canary and you can get it from here.

Essentially the pseudo code for my script goes as follows..

1. launch Chrome in headless mode (no visible window)
2. load a page and wait until the page is loaded
3. then wait a little longer until all the JS on the page has completed
4. then click on something using a querySelector to change the language to English
5. then grab all the source code from the page
6. then load that source code into cheerio and perform queries as needed

The full source code for this is below. You’ll of course need to install the 3 packages too.

> yarn add chrome-remote-interface chrome-launcher cheerio

I hope that helps some people! It took me ages to finally get to this place where I could scrape a dynamically generated webpage from Node.js.

So I recently had a job where we needed to remove the PayPal payment gateway in the Shopify checkout for particular products.

So a simple way for the client to manage this, is for them to tag those products they don’t want to be purchasable by PayPal, and then for us to write a script that detects those tags for those products in the checkout, and then removes that payment gateway.

Now it’s not possible to modify the checkout experience for standard Shopify stores. But if you’re a Shopify Plus customer then you can use the Script Editor App which does allow you to programatically modify the checkout experience. The API for this is here.

Now my language of expertise is JavaScript, and to use the Script Editor you have to use Ruby. This wasn’t actually too bad to learn. As someone that has done a lot of programming, looking up the syntax online for how to do something was quite easy. Particularly with the help of some Ruby pros in slack chats.

So, back to the problem at hand. If the client just added a tag called ‘no-paypal’ to whatever product they wanted to trigger the removal of the PayPal payment gateway, this is how we did it:

A gotcha I had trouble figuring out on my own: when you create a new script in the Script Editor, be sure to choose the correct script template. This is because the different types of scripts have different data objects available to them at runtime. In this case, I thus needed to choose the “Payment gateways” template.

Any questions/comments, let me know!

I had a project where I needed to setup a server that communicated over the SOAP protocol. I use Node.js for my backend server, so I began looking for libraries to use to work with.

I settled on strong-soap, but began to have issues with doing simple changes to the XML like editing the prefix for the namespace. I opened an issue for this, and there didn’t seem to be an easy way to remedy it.

It turned out the easiest and most robust solution was to explicitly write out my own XML and use request-promise-native to do a POST request with that custom crafted XML as the body to the server I was trying to communicate with.

Using ES6 I could use string interpolation to insert the values into the XML from the express server I set up to wrap around the SOAP request.

So essentially I could make POST requests to my server using JSON, and get a JSON response. And voilà, a JSON wrapped interface for a SOAP protocol.

A somewhat cut-down version of my actual code (that includes setting up https as it seems ever more important) is this…

The main advantages to this approach are:

• No extra libraries needed
• I can be 100% confident in what XML I am actually sending to the server. I had the SOAP 3rd party asking for the exact XML fragment in order to help troubleshoot issues with me. Using strong-soap I wasn’t sure as it was generated.

Any questions or thoughts let me know! Thanks.