Community Activity
- 2 Views
- 0 replies
- 0 kudos
'dbutils.jobs.taskValues.get' taking debug value in workflow, instead of actual value being set
Hi,I am trying to pass and set values from one wheel into another wheel in databricks workflow.I have used 'dbutils.jobs.taskValues.get' and 'dbutils.jobs.taskValues.set'I have used 'dbutils.jobs.taskValues.get' in second task and made sure to keep d...
- 2 Views
- 0 replies
- 0 kudos
- 46 Views
- 2 replies
- 1 kudos
- 46 Views
- 2 replies
- 1 kudos
- 1 kudos
Azure provides an Activity Log that records all write operations (PUT, POST, DELETE) for your storage account. This log includes information about who initiated the operation and when it occurred.
- 1 kudos
- 132 Views
- 1 replies
- 4 kudos
DLT Apply Changes Tables corrupt
Got a weird DLT error.Test harness using the new(ish) 'Apply Changes from Snapshot' Functionality and DLT Serverless (Current Channel). Azure Aus East Region.Has been working for several months without issue - but within the last week these DLT table...
- 132 Views
- 1 replies
- 4 kudos
- 19 Views
- 0 replies
- 0 kudos
Limited concurrent running DLT's within a pipeline
Hi Champions!We are currently experiencing a few unexplainable limitations when executing pipelines with > 50 DLT tables. It looks like, that there is some calculation in the background in place, to determine the maximum number of concurrent running ...
- 19 Views
- 0 replies
- 0 kudos
- 603 Views
- 2 replies
- 0 kudos
Community Edition workspace not found
Suddenly got logout from my account in the Community Edition. When I tried to login again, I received this error message: "We were not able to find a Community Edition workspace with this email. Please login to accounts.cloud.databricks.com to find t...
- 603 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @ledsouza and @vinitptdr! You can find helpful resources for Databricks Community Edition here. If the available resource doesn’t resolve your concern, please submit a ticket with the Databricks Support team for further assistance. Thank you.
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
How to specify path while creating tables using DLT
Hi All,I am trying to create table using DLT and would like to specify the path where all the files should reside.I am trying something like this:dlt.create_streaming_table( name="test", schema="""product_id STRING NOT NULL PRIMARY KEY, ...
- 32 Views
- 0 replies
- 0 kudos
- 67 Views
- 3 replies
- 0 kudos
minor bug in Lab 2 of ML Academy
# Remove listings with MentHlth > -40 should be # Remove listings with MentHlth < -40
- 67 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @jonxu!Could you please provide more details about the course and the specific Lab you are referring to?
- 0 kudos
- 270 Views
- 8 replies
- 1 kudos
01_demo_setup error
HelloI was following "Demo: Creating and Working with a Delta Table"while I have a community edition user.The first command in the Notebook is: %run ./setup/01_demo_setup But I got the following error:Notebook not found: Users/<my-email-was-here..>/s...
- 270 Views
- 8 replies
- 1 kudos
- 1 kudos
Still facing it myself. Any solution ?
- 1 kudos
- 14 Views
- 0 replies
- 0 kudos
Identity column and impact on performance
Hi,I want to define identity column in the Delta table.Based on documentation:"Declaring an identity column on a Delta table disables concurrent transactions. Only use identity columns in use cases where concurrent writes to the target table are not ...
- 14 Views
- 0 replies
- 0 kudos
- 99 Views
- 3 replies
- 4 kudos
Drive memory utilization cleanup
Issue Summary:When running multiple jobs on the same compute cluster, over time, I see an increase in memory utilization that is seemingly never fully released, even when jobs finish. This eventually leads to some jobs stalling out as memory hits the...
- 99 Views
- 3 replies
- 4 kudos
- 4 kudos
I'm encountering something similar. Immediately upon starting a cluster and triggering a job run, my memory usage jumps from 0 to about 20GB used and 15GB cached (see the attached screenshot). The data I am working with should be very small (less tha...
- 4 kudos
- 7 Views
- 0 replies
- 0 kudos
Python udf to pyspark udf conversion
Hi,I want to convert my python udf to pyspark udf, is there any guide/article on suggesting the best practices and avoid miscalculations if any
- 7 Views
- 0 replies
- 0 kudos
- 19 Views
- 2 replies
- 2 kudos
Working with Pandas through Abfss
Hi, I am unable to read and write pandas datframe through abfss protocol , Is there a work around for this , I do not want to store my files in dbfs
- 19 Views
- 2 replies
- 2 kudos
- 2 kudos
you can use volumes and mount the abfss location to unity catalog and can access the files present in azure in databricks. Regards,Avinash N
- 2 kudos
- 57 Views
- 3 replies
- 3 kudos
Lakehouse Federation
I use Lakehouse Federation to connect hive metastore (local VM) with metastore database type mysqlIt's can see database and table in hive but when I query data session continue running without failed or success.Do I have to migrate data to ADLs which...
- 57 Views
- 3 replies
- 3 kudos
- 3 kudos
As for Lakehouse Federation, only some sources are supported as of now. In that, connecting to HiveMetastore data which is in local VM is not supported, so you can migrate that data to ADLS and mount that as external location in unity catalog and que...
- 3 kudos
- 23 Views
- 1 replies
- 0 kudos
Identifying Full Refresh vs. Incremental Runs in Delta Live Tables
Hello Community,I am working with a Delta Live Tables (DLT) pipeline that primarily operates in incremental mode. However, there are specific scenarios where I need to perform a full refresh of the pipeline. I am looking for an efficient and reliable...
- 23 Views
- 1 replies
- 0 kudos
- 0 kudos
@yvishal519 wrote:Hello Community,I am working with a Delta Live Tables (DLT) pipeline that primarily operates in incremental mode. However, there are specific scenarios where I need to perform a full refresh of the pipeline. I am looking for an effi...
- 0 kudos
- 16 Views
- 1 replies
- 0 kudos
Driver log storage location
What directory would the driver log normally be stored in? Is it DBFS?
- 16 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ampammam ,The default storage location for driver logs in Databricks is on the local disk of the driver node. I think you can change this behavior going to cluster settings: Advanced Options→Logging→Destination
- 0 kudos