- 8164 Views
- 5 replies
- 1 kudos
Resolved! Delta External table
Hi I am new to databricks and need some inputs.I am trying to create Delta External table in databricks using existing path which contains csv files.What i observed is below code will create EXTERNAL table but provider is CSV.------------------------...
- 8164 Views
- 5 replies
- 1 kudos
- 1 kudos
@tajinder123 - can you please modify the syntax as below to create as a delta table CREATE TABLE employee123 USING DELTA LOCATION '/path/to/existing/delta/files';
- 1 kudos
- 5208 Views
- 1 replies
- 1 kudos
source set to GIT for Databricks Asset Bundle notebook_task - git authentication fails on run
My post was marked as Spam after trying to post the description of my issue so now I have posted the question on stackoverflow.
- 5208 Views
- 1 replies
- 1 kudos
- 14154 Views
- 0 replies
- 1 kudos
Notebook Detached java.net.SocketTimeoutException: Connect Timeout
When I attach a notebook to my cluster and run a cell the notebook is detached.Cell execution states:Waiting for compute to be readyThen the attached message is shown.Notebook detachedException execution context: java.net.SocketTimeoutException: Conn...
- 14154 Views
- 0 replies
- 1 kudos
- 2557 Views
- 1 replies
- 0 kudos
DLT SQL demo pipeline issue
Hi, First foray into DLT and following code exerts from the sample-DLT-notebook.I'm creating a notebook with the SQL below:CREATE STREAMING LIVE TABLE sales_orders_rawCOMMENT "The raw sales orders, ingested from /databricks-datasets."TBLPROPERTIES ...
- 2557 Views
- 1 replies
- 0 kudos
- 0 kudos
If you change the notebook default language as opposed to using magic command. I normally have it set to Python, I've wrongly assumed DLT would transpose as can't use magic command but have to change default in order for it to work.
- 0 kudos
- 2325 Views
- 1 replies
- 0 kudos
Validating Dlt Pipeline
Whenever I try validating a pipeline that already runs productively without any issue, it throws me the following error:BAD_REQUEST: Failed to load notebook '/Repos/(...).sql'. Only SQL and Python notebooks are supported currently.
- 2325 Views
- 1 replies
- 0 kudos
- 1692 Views
- 0 replies
- 0 kudos
Download event and run logs
how can I download the run and event logs? spark UI is loading them from somewhere but I couldn't find them in dbfs nor on s3
- 1692 Views
- 0 replies
- 0 kudos
- 1431 Views
- 0 replies
- 0 kudos
Operation on target Databricks_Transformation_Operation failed: Invalid access token.
Seeking efficient strategies for re solving errors in Azure Data Factory pipelines. Any valuable tips or techniques?
- 1431 Views
- 0 replies
- 0 kudos
- 8349 Views
- 5 replies
- 3 kudos
Resolved! Databricks runtime 14.3 gives error scala.math.BigInt cannot be cast to java.lang.Integer
We have a cluster running on 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12).We want to test with a different type of cluster (14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12))And all of a sudden we get errors that complain about a casting a Big...
- 8349 Views
- 5 replies
- 3 kudos
- 3 kudos
I have logged the issue with Microsoft last week and they confirmed it is a Databricks bug. A fix is supposedly being rolled out at the moment across Databricks regions. As anticipated, we have engaged the Databricks core team to further investigate ...
- 3 kudos
- 1823 Views
- 0 replies
- 0 kudos
broadcasted table reuse
in spark, table1 is small and broadcasted and joined with table 2. output is stored in df1. again, table1 is required to join with table3 and output need to be stored in df2. do it need to be broadcasted again?
- 1823 Views
- 0 replies
- 0 kudos
- 2174 Views
- 0 replies
- 0 kudos
how to run a group of cells in databricks ?
Hello,I was experimenting with a ML model with different parameters and check the results. However, the important part of this code is contained in a couple of cells (say cell # 12, 13 & 14). I like to proceed to the next cell only when the results a...
- 2174 Views
- 0 replies
- 0 kudos
- 3775 Views
- 1 replies
- 0 kudos
Unable to read data from API due to Private IP Restriction
I have data in my API Endpoint but am unable to read it using Databricks. My data is limited to my private IP address and can only be accessed over a VPN connection. I can't read data into Databricks as a result of this. I can obtain the data in VS C...
- 3775 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi AravindNaniThis is more of infrastructure questions, you have to make sure that:1) Your databricks Workspace is provisioned in VNET Injection mode2) Your VNET is either peered to "HUB" network where you have S2S VPN Connection to API or you have t...
- 0 kudos
- 3721 Views
- 1 replies
- 0 kudos
using the api for getting cost in usd
I'm trying to use the API of billable usage and I do get a report but I have not been able to get the usd cost report, only the dbuHours. I guess I've to change the meter_name but I cannot find the key for that parameter anywhere
- 3721 Views
- 1 replies
- 0 kudos
- 5057 Views
- 1 replies
- 0 kudos
databricks email notification
In databricks, if a job fails, then an email is sent off as notification.The recipient, receives the email with the link to the databricks workspace.Question:How is it possible the email is sent without any link, just the plain text in the email is w...
- 5057 Views
- 1 replies
- 0 kudos
- 2181 Views
- 0 replies
- 0 kudos
set up Azure Databricks workspace and Unity catalog - how to automate not using Terraform
Hi everyone, I am looking for a way to automate initial setup of Azure Databricks workspace and Unity Catalog but can't find anything on this topic other than Terraform. Can you share if this is possible with powershell, for example. Thank you un adv...
- 2181 Views
- 0 replies
- 0 kudos
- 10497 Views
- 3 replies
- 1 kudos
Fuzzy Match on PySpark using UDF/Pandas UDF
I'm trying to do fuzzy matching on two dataframes by cross joining them and then using a udf for my fuzzy matching. But using both python udf and pandas udf its either very slow or I get an error. @pandas_udf("int")def core_match_processor(s1: pd.Ser...
- 10497 Views
- 3 replies
- 1 kudos
- 1 kudos
I'm now getting the error: (SQL_GROUPED_AGG_PANDAS_UDF) is not supported on clusters in Shared access mode.Even though this article clearly states that pandas udf is supported for shared cluster in databrickshttps://www.databricks.com/blog/shared-clu...
- 1 kudos
