- 3493 Views
- 3 replies
- 0 kudos
Unable to overwrite table to Azure sql db
Hi I have a requirement to read table from azure sql db and update the table in azure databricks with transformations and overwrite updated table to the azure sql db but due to lazy evaluation of pyspark im unable to overwrite the table in azure sql ...
- 3493 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Kingston Make sure that you have the proper permissions on the SQL server for the user you do the authentication through JDBC with, i.e. database reader / database writer. Then your approach can go in two directions, push the data from Databrick...
- 0 kudos
- 2174 Views
- 0 replies
- 0 kudos
Connect databricks
I discovered recently mlflow managed by Databricks so I'm very new to this and I need some help.Can someone explain for me clearly the steps to do to be able to track my runs into the Databricks API.Here are the steps I followed :1/ Installing Databr...
- 2174 Views
- 0 replies
- 0 kudos
- 4168 Views
- 0 replies
- 0 kudos
Unable to write to S3 bucket from Databricks using boto3
I am unable to write data from Databricks into an S3 bucket. I have set up the permissions both on the bucket policy level, and the user level as well (Put, List, and others are added, have also tried with s3*). Bucket region and workspace region are...
- 4168 Views
- 0 replies
- 0 kudos
- 5237 Views
- 2 replies
- 0 kudos
Accessing data from a legacy hive metastore workspace on a new Unity Catalog workspace
Hello,For the purposes of testing I'm interested in creating a new workspace with Unity Catalog enabled, and from there I'd like to access (external - S3) tables on an existing legacy hive metastore workspace (not UC enabled). The goal is for both wo...
- 5237 Views
- 2 replies
- 0 kudos
- 0 kudos
Your aim is to access external S3 tables from a Unity Catalog workspace without data duplication and keeping data updates synchronized. Configure external location permissions. This ensure that both your Unity Catalog and Hive metastore workspaces h...
- 0 kudos
- 5090 Views
- 2 replies
- 0 kudos
Select from a dynamic table name returned by databricks function
I have a databricks function that returns a table_nameCREATE OR REPLACE FUNCTION test_func() RETURNS string READS SQL DATA RETURN 'table_name'I want to select from the table that is returned by this function. How can I make it work in SQL, some...
- 5090 Views
- 2 replies
- 0 kudos
- 0 kudos
I looked at this posthttps://stackoverflow.com/questions/77475436/in-databricks-workbook-using-spark-sql-how-to-pass-parameters-thru-sql-udf-func What I want is to replace the static table name with the table name passed as parameter (param_table_nam...
- 0 kudos
- 1759 Views
- 0 replies
- 0 kudos
Self Bootstrap Failure Community Edition
Trying to start a new compute this morning and I get the "Self Bootstrap Error." I saw some people with similar error but their fixes involved DNS settings. I've haven't made any dns changes and was working fine without any problem last night. I real...
- 1759 Views
- 0 replies
- 0 kudos
- 6642 Views
- 3 replies
- 0 kudos
Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise
I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ...
- 6642 Views
- 3 replies
- 0 kudos
- 0 kudos
I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, makin...
- 0 kudos
- 12423 Views
- 4 replies
- 1 kudos
Dynamic Number of Tasks in Databricks Workflow
Do Databricks workflows support creating a workflow with a dynamic number of tasks?For example, let's say we have a DAG like this:T1 -> T2(1) -> T2(2) -> ..... -> T3 T2(n-1) -> T2(n)...
- 12423 Views
- 4 replies
- 1 kudos
- 1 kudos
@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | MediumOne possibility is after running earlier job, process the output to create a dynamic number of tasks in s...
- 1 kudos
- 3034 Views
- 1 replies
- 0 kudos
How to access Spark UI metrics in an automated way (API)
I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from loca...
- 3034 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.
- 0 kudos
- 6736 Views
- 7 replies
- 0 kudos
Trouble on Accessing Azure Storage from Databricks (Python)
I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient pr...
- 6736 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi,you can find storage account firewall information by accessing resource in azure portal Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Lo...
- 0 kudos
- 4531 Views
- 6 replies
- 0 kudos
can't execute the code
When I was executing the code, I was getting this error:"Notebook detached×Exception when creating execution context: java.net.SocketTimeoutException: Connect Timeout"Can someone help me?
- 4531 Views
- 6 replies
- 0 kudos
- 1790 Views
- 0 replies
- 0 kudos
raise Py4JJavaError while changing data type of a column
HiI'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format2/18/2021 7:20:12 PMSo I wrote following commandfrom py...
- 1790 Views
- 0 replies
- 0 kudos
- 4304 Views
- 3 replies
- 0 kudos
Query example for databricks Query History API
Hi I am trying to get query history data from my SQL warehouse. Following previous examples is not working. databricks_workspace_url = "xxx"token = "xxx"start_time = 1707091200end_time = 1707174000api_endpoint = f"{databricks_workspace_url}/api/2.0/s...
- 4304 Views
- 3 replies
- 0 kudos
- 0 kudos
@Cheryl - you can use query_start_time=2023-01-01T00:00:00Z as a parameter to filter for the time frame. available filter criteria are given below - https://docs.databricks.com/api/workspace/queryhistory/list#filter_by-query_start_time_range
- 0 kudos
- 1895 Views
- 0 replies
- 0 kudos
CLI: Cannot Configure Additional Profiles in web terminal
Hello,I'm having a hard adding secrets since the change from the legacy CLI. Executing "databricks configure" as referenced in the personal access token section of this document does nothing. I have verified that the CLI is installed as v0.216.0.
- 1895 Views
- 0 replies
- 0 kudos
- 2963 Views
- 1 replies
- 0 kudos
Traversing to previous rows and getting the data based on condition
Sample Input data setClusterIdEventEventTime1212-18-r9u1kzn1RUNNING2024-02-02T11:38:30.168+00:001212-18-r9u1kzn1TERMINATING2024-02-02T13:43:33.933+00:001212-18-r9u1kzn1STARTING2024-02-02T15:50:05.174+00:001212-18-r9u1kzn1RUNNING2024-02-02T15:54:21.51...
- 2963 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 517 | |
| 349 | |
| 287 |
