Scraper

Remember when we made raised events in Event Hub while we were making our favorite color API (Spewer)? Now let’s do things with those events we’ve been raising! We are going to make a scraper, which “scrapes” events from event hub and logs them so we have a record of what is happening.

Let’s start by grabbing those events. First we are going to need to make a webhook, so let’s go the Event Hub in WSO2.

As we put in all the information, you may notice that we need an endpoint.

We are going to use the endpoint from the API gateway, so let’s look the up. Go to AWS API Gateway, find your gateway you made in the previous lesson, go to stages and click dev to find your URL.

Once you know your URL, use the “Try it Out” feature to make your webhook.

Now, let’s go to the code of our lambda and update it to get the information. We are going to need the ‘aws-sdk’ module, but luckily Lambda imports that for us so we don’t need to use npm to import it.

Now, let’s put some params in the AWS Parameter Store so we can get them. This time instead of doing multiple strings, let’s just save an object in the parameter store. We are going to need the parameters for the oracle database and the security key for the web hook, so let’s save that all now. write an object that looks like this (but uses your information)-

You can find your security key in WSO2 using a GET request with the try it out feature-

Go to the parameter store and save your object.

Now, write a function to get and parse those parameters-

Then let’s write a program to make sure that the call we get is valid.

(You are going to need to declare params globally, I forgot to put that in the picture, but on line 4 use

)

Line 7 is kind of interesting, and involves understanding how lambdas work. When the API get called, a new container(a type of virtual machine) is spun up and the function is run. Then the state that the lambda is in frozen and will be used again if the lambda is invoked within a short period of time. This means that the first time a lambda is called it can be very slow, but then the subsequent ones are faster for a time. line 7 says that if the lambda already has your params it doesn’t need to spend time (and therefore money, because lambs are billed by the millisecond) to grab the params again. We will take advantage of this big time wen we are making oracle database connections later.

Push your changes to Git Hub. Then use Postman to call your favorite color API and change your favorite color, which will raise and event, which will invoke your lambda to test it out. You can make sure everything is working on AWS CloudWatch.

Once you know everything is working, let’s set up a table in SQL Developer on the CESDEV database-

Then let’s import the “oracledb” npm module.

Run “npm init” and set it up using the Apache-2.0 license.

Then run “npm install oracledb”.

Now, here comes the interesting part. Remember how we need a library so that the oracledb npm module works? Well, the library has some problems with lambda. Luckily, Gary Crye put together a library that works. Use “git clone https://github.com/byu-oit/node-oracle-lambda.git” and copy the lib folder from gary’s repository into your folder.

Sweet, with that, let’s finish up our lambda.

Let’s set up oracle connections and import crypto-

Then, let’s create a connection globally (so it can be used between invocations of the lambda) and a sql statement to write into our table.

This website will explain a little more about the whole connections thing https://www.jeremydaly.com/reuse-database-connections-aws-lambda/.

Set it up to use the connections.

Then let’s write the code to write to the table.

Now everything should work. Postman to change your name a few times and then go to your table and see if it recorded.

If it works, then you are done!