cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

dm7
by New Contributor II
  • 1085 Views
  • 0 replies
  • 0 kudos

DLT CDC/SCD - Taking the latest ID per day

Hi I'm creating a DLT pipeline which uses DLT CDC to implement SCD Type 1 to take the latest record using a datetime column which works with no issues:@dlt.view def users(): return spark.readStream.table("source_table") dlt.create_streaming_table(...

  • 1085 Views
  • 0 replies
  • 0 kudos
nikhilmb
by New Contributor II
  • 2146 Views
  • 3 replies
  • 0 kudos

Error ingesting zip files: ExecutorLostFailure Reason: Command exited with code 50

Hi,We are trying to ingest zip files into Azure Databricks delta lake using COPY INTO command. There are 100+ zip files with average size of ~300MB each.Cluster configuration:1 driver: 56GB, 16 cores2-8 workers: 32GB, 8 cores (each). Autoscaling enab...

  • 2146 Views
  • 3 replies
  • 0 kudos
Latest Reply
nikhilmb
New Contributor II
  • 0 kudos

Although we were able to copy the zip files onto the DB volume, we were not able to share them with any system outside of the Databricks environment. Guess delta sharing does not support sharing files that are on UC volumes.

  • 0 kudos
2 More Replies
FelipeRegis
by New Contributor
  • 1577 Views
  • 0 replies
  • 0 kudos

Not able to access data registered in Unity Catalog using Simba ODBC driver

Hi folks, I'm working on a project with Databricks using Unity Catalog and a connection to SSIS (SQL Server Integration Services).My team is trying to access data registered in Unity Catalog using Simba ODBC driver version 2.8.0.1002. They mentioned ...

odbc_driver_unity_catalog_issue.png
  • 1577 Views
  • 0 replies
  • 0 kudos
sharma_kamal
by New Contributor III
  • 4550 Views
  • 3 replies
  • 4 kudos

Resolved! Unable to read available file after downloading

After downloading a file using `wget`, I'm attempting to read it by spark.read.json.I am getting error: PATH_NOT_FOUND - Path does not exist: dbfs:/tmp/data.json. SQLSTATE: 42K03File <command-3327713714752929>, line 2 I have checked the file do exist...

  • 4550 Views
  • 3 replies
  • 4 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 4 kudos

Hi @sharma_kamal , Good Day!  Could you please try the below code suggested by @ThomazRossito , it will help you.  Also please refer to the below document to work with the files on Databricks:  https://docs.databricks.com/en/files/index.html Please l...

  • 4 kudos
2 More Replies
MohsenJ
by Contributor
  • 2135 Views
  • 0 replies
  • 0 kudos

the inference table table doesn't get updated

I setup a model serving endpoint and created a monitoring dashboard to monitor its performance. The problem is my inference table doesn't get updated by model serving endpoints. To test the endpoint I use the following codeimport random import time ...

  • 2135 Views
  • 0 replies
  • 0 kudos
arkiboys
by Contributor
  • 7294 Views
  • 2 replies
  • 0 kudos

reading mount points

Hello,Previously I was able to run the folowing command in databricks to see a list of the mount points but it seems the system does not accept this anymore as I get the following error.Any thoughts on how to get a list of the mount points?Thank youd...

  • 7294 Views
  • 2 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello,Previously I was able to run the folowing command in databricks to see a list of the mount points but it seems the system does not accept this anymore as I get the following error.Any thoughts on how to get a list of the mount points?Thank youd...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
DataYoga
by New Contributor
  • 2800 Views
  • 1 replies
  • 0 kudos

Informatica ETLs

I'm delving into the challenges of ETL transformations, particularly moving from traditional platforms like Informatica to Databricks. Given the complexity of legacy ETLs, I'm curious about the approaches others have taken to integrate these with Dat...

  • 2800 Views
  • 1 replies
  • 0 kudos
Latest Reply
owene
New Contributor II
  • 0 kudos

@Retired_mod Do you know if Informatica Cloud Modernization can convert mappings into Delta Live Tables? Do we have to use Informatica Cloud for this, or can we use it as a one time migration and maintain the artifacts in DataBricks?Alternatively, we...

  • 0 kudos
Chrispy
by New Contributor
  • 2927 Views
  • 0 replies
  • 0 kudos

Spark Handling White Space as NULL

I have a very strange thing happening.  I'm importing a csv file and nulls and blanks are being interpreted correctly.  What is strange is that a column that regularly has a single space character value is having the single space converted to null.I'...

  • 2927 Views
  • 0 replies
  • 0 kudos
Nandhini_Kumar
by New Contributor III
  • 1166 Views
  • 1 replies
  • 0 kudos

How to identify Worker and Driver instance in AWS console for databricks instance?

For AWS databricks, i have configured 1 worker and 1 driver node with same node type. In the AWS console, all details are same for the two instance, as instance id only different. How to identify which instance id is for worker and which one is for d...

  • 1166 Views
  • 1 replies
  • 0 kudos
Latest Reply
Nandhini_Kumar
New Contributor III
  • 0 kudos

Hi @Retired_mod Thanks for your response.There is no instance pool ID while configured the cluster, then how will i able to differentiate then.Could you give any alternative way for finding the driver instance id and worker instance id in the AWS con...

  • 0 kudos
Azsdc
by New Contributor
  • 2144 Views
  • 0 replies
  • 0 kudos

Usage of if else condition for data check

Hi,In a particular Workflows Job, I am trying to add some data checks in between each task by using If else statement. I used following statement in a notebook to call parameter in if else condition to check logic.{"job_id": XXXXX,"notebook_params": ...

  • 2144 Views
  • 0 replies
  • 0 kudos
Ha2001
by New Contributor
  • 1933 Views
  • 1 replies
  • 1 kudos

Databricks Repos API Limitations

Hi, I have started using databricks recently, and I'm not able find a right solution in the documentations. i have linked multiple repos in my databricks workspace in the repos folders, and I wanted to update the repos with remote AzureDevops reposit...

Community Platform Discussions
azure devops
Databricks
REST API
  • 1933 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 1 kudos

Hi @Ha2001 , Good Day!  Databricks API has a limit of 10 per second for the /repos/* combined requests in the workspace. You can check the below documentation for the API limit:  https://docs.databricks.com/en/resources/limits.html#:~:text=Git%20fold...

  • 1 kudos
jcozar
by Contributor
  • 3236 Views
  • 4 replies
  • 1 kudos

Resolved! Spark streaming query stops after code exception in notebook since 14.3

Hi!I am experiencing something that I cannot find in the documentation: in databricks, using the databricks runtime 13.X, when I start a streaming query (using .start method), it creates a new query and while it is running I can execute other code in...

  • 3236 Views
  • 4 replies
  • 1 kudos
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

You can use the help portal: https://help.databricks.com/s/

  • 1 kudos
3 More Replies
GlennStrycker2
by New Contributor III
  • 1091 Views
  • 1 replies
  • 1 kudos

Why so many different domains and accounts?

I've lost count of how many different domains and accounts Databricks is requiring for me to use their services.  Every domain is requiring its own account username, password, etc., and nothing is synced.  I can't even keep track of which email addre...

  • 1091 Views
  • 1 replies
  • 1 kudos
Latest Reply
GlennStrycker2
New Contributor III
  • 1 kudos

Pluscustomer-academy.databricks.comaccounts.cloud.databricks.comdatabricks.my.site.com

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors