Microsoft Fabric: Importing Data from External APIs Made Easy

Microsoft Fabric: Importing Data from External APIs Made Easy

Step-by-step guide to linking external APIs with Microsoft Fabric and moving data to Data Lakehouse

Arun R Nair's photo
Arun R Nair
·

3 min read

Pre-requisites

Microsoft Fabric account with proper license

API with key

Steps

  • Open Microsoft fabric and navigate to the Workspaces.

  • You can use the default workspace (My workspace) or create a new workspace. Here, I'm using the default workspace.

    Fig 1

  • Create a Lakehouse to store the data.

    Fig 2

  • Once the Lakehouse is created, you will see two default folders: Tables and Files. Tables will store the lakehouse tables, and Files will store the lakehouse files.

    Fig 3

  • Create a subfolder to store the extracted files. My folder name is NewsHeadlines.

  • Select Data Factory component from the list and create a Data Pipeline.

    Fig 4

  • My pipeline name is API_Integration

    Fig 5

  • Open the Pipeline and click Pipeline activity to start with a blank canvas.

    Fig 6

  • Select 'Copy Data' to create the pipeline.

    Fig 7

  • Configure Source and Destination for the copy activity
    Rename the activity name.

  • Choose one API for the integration.
    Here I'm using https://newsapi.org/ for this exercise.

    My API is: https://newsapi.org/v2/top-headlines?country={country code}&category={category}&apiKey={api}
    This endpoint provides live top and breaking headlines for a country.

  • Source -> Connection -> Select -> More -> Search "Rest"

  • Select REST as the source
    Fill the 'Base Url'
    Select 'Create new connection'
    Give a connection name and click Connect

  • We should invoke the full API in the Relative URL cell

  • We can give the full URL in the Pipeline expression builder.

  • Click the Preview button to see the sample data.

    • Source is ready.

    • Now we can configure destination.

      Go to the destination tab and select our lakehouse from the drop down.

    • Navigate to our subfolder and choose the destination folder.

  • Now our pipeline is ready to run. Click the validate button to validate the pipeline.

  • Once everything is ready, click the Run button to execute the pipeline.

    Wow, the pipeline completed successfully. Now, check the destination folder to find the data.

  • We can parametrize the input values (Country, Category, and API). Go to the pipeline parameters and create new parameters.

  • To manage these parameters, we should change the expression.

  • Now, initiate the pipeline run using these parameters by giving new values.

    Integrating external APIs with Microsoft Fabric can significantly enhance your data processing capabilities. By following the outlined steps, you can efficiently set up a robust pipeline for data ingestion and storage. Ensure you validate and test your pipeline thoroughly to achieve seamless integration. With the flexibility to parameterize inputs, you can easily adapt the pipeline for various use cases, making your data workflows more dynamic and efficient.

MicrosoftfabricAzureData SciencedataengineeringAPIs