Since I didn't find a way of running SQL code directly from S3, I moved ahead by importing s3 file to Databricks using API by following below :

1. Added -- Databricks notebook source on top of each file so that it is treated as Databricks notebook. 

2. Created one notebook with logic to import the sql file to Databricks workspace from s3. 

The logic includes:

• Read s3 file using boto3
• Convert the content of the file to base64 encoding and then make it string.
• Pass the base64 encoded string in content 
  
  
• import_params = {
      "path": dbx_workspace_path,  # Replace with the target Databricks path
      "format": "SOURCE",
      "language": "SQL",
      "content": base64_string,
      "overwrite": True
  }

  Pass this import_params dictionary as payload as post request to api 

  api/2.0/workspace/import. 

  url will look like 

  url = f"https://{dbx_host}/api/2.0/workspace/import"

   

  Note: The dbx_workspace_path should already exist. If it doesn't then you can create it through UI or through API 

  api/2.0/workspace/mkdirs
  This can create full path at once. For example folder1/folder2/folder3. If all the 3 folders are not present and if you provide this path in payload, it will create all together in the given hierarchy.

   

  3. Once the notebook is available in Databricks Workspace, it can be ran via jobs or Databricks Jobs Api