Creating Import Connections

Web service import connections are managed with the other import connections on the Admin > Import Connections screen. In this example we’ll add a new REST Web Service.

There are three authentication types

• OAuth 2.0
• Anonymous
• Basic HTTP (Username/Password)

Anonymous Authentication

Anonymous authentication is the simplest and allows you to pull data from web services that don’t require authentication. You can find data sources for testing at
https://github.com/public-apis/public-apis.

In step 2 you can set the API Endpoint Base URL. This is an optional field that allows you to restrict what URLs you can visit with this connection. For example, if you used
https://spiderstrategies.zendesk.com/api/v2/
for the base URL, imports using this connection could pull data from
https://spiderstrategies.zendesk.com/api/v2/metrics
but they couldn’t pull data from
https://spiderstrategies.zendesk.com/something
. If you leave the Base URL blank, imports will be able to use any URL.

The final step is assigning owners who can use this import connection.

Basic HTTP (Username/Password) Authentication

Basic HTTP authentication sends a username and password to the API server.

Steps 2 and 3 for Basic authentication are exactly the same as Anonymous authentication. You choose an API Endpoint Base URL, and then assign ownership to the connection.

OAuth 2.0 Authentication

Initial communication

OAuth 2.0 is the most complicated authentication type because there are two parts. First you get an Access Token from the remote API's token server. Then you use that Access Token in all further communication with the API get the data you want.

The first step in the new import connection wizard is all about sending information to the remote API's token server in order to get back either an Access Token or a Refresh Token.

1. The HTTP Method is either GET or POST. If you’re unsure, stay with the default of GET.
2. The Access Token Endpoint URL is the URL of the server you’re communicating with to get the Access Token.
3. Specify any URL parameters here rather than in the query string. For example, rather than using an endpoint URL of
   https://test.com?var1=apple&var2=orange
   , you’d use an endpoint URL of
   https://test.com
   and set var1 and var2 as URL parameters. This automatically encodes the information you’re sending and allows other URL parameters to be added later.
4. It’s common to set HTTP Request Headers like Authentication and Accept.
5. The Request Body is only available when the HTTP Method is POST.

As you can see, REST web service connections have a lot of configuration options. That's because every API needs different information to be provided in different ways. Be sure to read the documentation for the API that you're connecting with to see exactly how to communicate with it.

There's also a good chance that you'll need to do some setup in the web service you're connecting to. For example you may need to set up a "client" or "app" and then provide a Client ID and/or Client Secret via one of the 5 settings above. Again, see the other API's documentation for details.

When you click Next on step 1, Spider Impact will attempt to get a token from the remote API token server, and will move you to the correct step of the wizard based on the type of the token you get back.

Receiving a refresh token

If the remote API’s token server responds with an Access Token, you'll skip this step and continue on to the Access Token step below. If the server responds with a Refresh Token, however, one extra step gets added to the wizard, bringing the total number of steps to 4.

Here we can see that we’ve received a Refresh Token, and that we need to tell Spider Impact how to send that Refresh Token back to the token server in the future. This will often be in the same way as the original request, so those settings are automatically copied over from step 1.

If everything is successful when you click Next, the remote API token server will respond with an Access Token and we'll continue to the next step.

Receiving an access token

Here we see we're on step 3 of 4, but this is the same as step 2 of 3 when refresh tokens aren’t involved. At this point we have an Access Token, and we need to tell Spider Impact how to send that Access Token to the API in the future. This step is all about choosing what data you want to send along with every future request to the API.

The final step is assigning owners who can use this import connection, which is the same for every type of import connection.

Creating Imports

To import data from a Web Service, choose one of the web service data sources when creating the import. In this example we’re creating a dataset and are choosing REST Web Service.

Because we’ve already specified all of the authentication information when we set up the import connection, all we have to do is add ticket_metrics to the end of the API URL. We’ll try to advance to the next step by clicking Next.

In this example there’s an error getting data from the server. When that happens, Spider Impact will show a second-level dialog that contains the error message and debugging info.

Setting up a web service import is often a back-and-forth process, regardless of which two systems you’re connecting. You’ll ask the server for information and it’s either not quite right, or there’s an error that needs debugging.

Spider Impact does its best to make this debugging as simple as possible. In this example it’s saying we need to ask the server for JSON data.

So, we’ll set an Accept request header with a value of application/json and we’ll try again.

Now when we click Next, we successfully advance to step 2 where we see all of the data returned from the API. Just because you see data, however, doesn’t always mean it’s the right data. If you still want to debug, just click the Communication Details button.

That same second-level dialog opens showing you all of the results from the server.

After closing that dialog, we’re back on step 2 of the new import. We could click the dialog’s Back button to change what we request, but this is exactly what we want, so it’s time to choose our data.

Step 2 is all about choosing the repeating blocks of the data you want. For example, if we wanted just the users, we could select this block, and it would automatically select all other user blocks just like it.

Instead, we’re going to select the high-level objects in the array that represent individual issues.

When we click Next, everything from here on is the same as importing data from any source.  We go to the transformation step, and then on to choosing which fields you want to create.

We’ll schedule the import to run every night, and now we have a dataset with our top 100 issues that automatically updates every night.

Debugging with Curl

All modern operating systems (Windows 10+, macOS, Linux, etc.) ship with a command line utility called curl that is very helpful for debugging web service connections. Curl sends information to web servers and shows you the results. If you're having trouble setting up a web service import in Spider Impact, it can be helpful to get Impact out of the loop and just use curl to see exactly what you're sending and receiving.

For example, the Zendesk Sell API says that you can get information about your Zendesk Sell user from the API endpoint https://api.getbase.com/v2/users/self. And, the API instructions say you need to set an Accept HTTP header with a value of application/json. So, we'll hit that endpoint via curl by typing this into Terminal (or whatever command line utility our operating system has):

curl https://api.getbase.com/v2/users/self \
-H "Accept: application/json"

The server's response is:

{"errors":[{"error":{"code":"unauthorized", "details":"Required access token is missing, malformed, expired, or invalid.", "message":"request is unauthorized"}, "meta": {"type": "error","links": {"more_info": "https://developers.getbase.com/docs/rest/articles/errors"}}}], "meta": {"type": "collection","logref": "8130defa5ced05ec-IAD", "links": {"more_info": "https://developers.getbase.com/docs/rest/articles/errors"}, "http_status": "401 Unauthorized"}}(base)

It says "Required access token is missing". When we double-check the API documentation, sure enough, it does say we need to provide a token that we get from Zendesk Sell. So we'll try again with this:

curl https://api.getbase.com/v2/users/self \
-H "Accept: application/json" \
-H "Authorization: Bearer 354b14963be05431509c15b18e2bbfc558edf7fbd2cb42f72bb7bc6698c2"

This time the server returns:

{"data":{"id":15606,"name":"Dev Team","email":"test@spiderstrategies.com","phone_number":null,"timezone":null,"confirmed":true,"invited":true,"status":"active","role":"admin","roles":[],"reports_to":null,"team_name":null,"group":null,"detached":false,"identity_type":"person","system_tags":[],"zendesk_user_id":"118945","sell_login_disabled":true,"updated_at":"2023-09-13T16:48:47Z","created_at":"2019-11-04T14:27:50Z","deleted_at":null},"meta":{"type":"user"}}(base)

Success! Now we can confidently create an import based on connection data that we know works.