cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
ashraf1395
by • Valued Contributor III
  • 670 Views
  • 1 replies
  • 0 kudos

Schema issue while fetching data from oracle

I dont have the complete context of the issue.But Here it is what I know, a friend of mine facing this""I am fetching data from Oracle data in databricks using python.But every time i do it the schema gets changesso if the column is of type decimal f...

Data Engineering
Reply
  • 670 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Thanks for your question!To address schema issues when fetching Oracle data in Databricks, use JDBC schema inference to define data types programmatically or batch-cast columns dynamically after loading. For performance, enable predicate pushdown and...

  • 0 kudos
chris_b
by • New Contributor
  • 655 Views
  • 1 replies
  • 0 kudos

Increase Stack Size for Python Subprocess

I need to increase the stack size (from the default of 16384) to run a subprocess that requires a larger stack size.I tried following this: https://community.databricks.com/t5/data-engineering/increase-stack-size-databricks/td-p/71492And this: https:...

Data Engineering
Reply
  • 655 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Thanks for your question!Are you referring to a Java stack size (-Xss) or a Python subprocess (ulimit -s)?

  • 0 kudos
upatint07
by • New Contributor II
  • 661 Views
  • 1 replies
  • 0 kudos

Facing Issue in "import dlt" using Databricks Runtime 14.3 LTS version

Facing issues while Importing dlt library in Databricks Runtime 14.3 LTS. Previously while using the Runtime 13.1 The `import dlt` was working fine but now when updating the Runtime version to 14.3 LTS it is giving me error. 

upatint07_0-1724733273085.png upatint07_1-1724733284564.png
Data Engineering
Reply
  • 661 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Thanks for your question! Unfortunately, this is actually a known limitation with Spark Connect clusters.

  • 0 kudos
CURIOUS_DE
by • New Contributor III
  • 665 Views
  • 1 replies
  • 1 kudos

A Surprise Findings in Delta Live Table

While DLT has some powerful features, I found myself doing a double-take when I realized it doesn’t natively support hard deletes. Instead, it leans on a delete flag identifier to manage these in the source table. A bit surprising for a tool of its c...

Data Engineering
Reply
  • 665 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 1 kudos

Thanks for your feedback ! I believe, Delta Live Tables (DLT) does not natively support hard deletes and instead uses a delete flag identifier to manage deletions, a design choice rooted in ensuring compliance with regulations like GDPR and CCPA. Thi...

  • 1 kudos
dener
by • New Contributor
  • 706 Views
  • 1 replies
  • 0 kudos

Infinity load execution

I am experiencing performance issues when loading a table with 50 million rows into Delta Lake on AWS using Databricks. Despite successfully handling other larger tables, this especific table/process takes hours and doesn't finish. Here's the command...

Data Engineering
Reply
  • 706 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Thank you for your question! To optimize your Delta Lake write process: Disable Overhead Options: Avoid overwriteSchema and mergeSchema unless necessary. Use: df.write.format("delta").mode("overwrite").save(sink) Increase Parallelism: Use repartition...

  • 0 kudos
alexgavrysh
by • New Contributor
  • 597 Views
  • 1 replies
  • 0 kudos

Job scheduled run fail alert

Hello,I have a job that should run every six hours. I need to set up an alert for the case if this doesn't start (for example, someone paused it). How do I configure such an alert using Databricks native alerts?Theoretically, this may be done using s...

Data Engineering
Reply
  • 597 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Thank you for your question! Here’s a concise workflow to set up an alert for missed job runs in Databricks: Write a Query: Use system tables to identify jobs that haven’t started on time.Save the Query: Save this query in Databricks SQL as a named q...

  • 0 kudos
Thor
by • New Contributor III
  • 707 Views
  • 1 replies
  • 0 kudos

Native code in Databricks clusters

Is it possible to install our own binaries (lib or exec) in Databricks clusters and use JNI to execute them?I guess that Photon is native code as far as I could read so it must use a similar technic.

Data Engineering
Reply
  • 707 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Thanks for your question! I believe it should be possible, although Photon itself is not extensible by users. Are you currently facing any issues while installing and using your own libraries, and JNI to execute them?

  • 0 kudos
ed_carv
by • New Contributor
  • 692 Views
  • 1 replies
  • 1 kudos

Databricks S3 Commit Service

Is Databricks S3 Commit Service enabled by default if Unity Catalog is not enabled and the compute resources run in our AWS account (classic compute plane)? If not, how can it be enabled?This service seems to resolve the limitations with multi-cluste...

Data Engineering
Reply
  • 692 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 1 kudos

  No, the Databricks S3 commit service is not guaranteed to be enabled by default in the AWS classic compute plane. The configuration may vary based on your specific workspace setup. How can it be enabled? To enable the Databricks S3 commit service, ...

  • 1 kudos
David_Billa
by • New Contributor III
  • 611 Views
  • 8 replies
  • 3 kudos

Extract datetime value from the file name

I've the filename as below and I want to extract the datetime values and convert to datetime data type. This_is_new_file_2024_12_06T11_00_49_AM.csvHere I want to extract only '2024_12_06T11_00_49' and convert to datetime value in new field. I tried S...

Data Engineering
Reply
  • 611 Views
  • 8 replies
  • 3 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 3 kudos

Unfortunately I am not able to make it work with SQL functions

  • 3 kudos
7 More Replies
peritus
by • New Contributor II
  • 868 Views
  • 5 replies
  • 5 kudos

Synchronize SQLServer tables to Databricks

I'm new to Databricks and, I'm looking to get data from an external database into Databricks and keep it synchronized when changes occur in the source tables. It seems like I may be able to some form of change data capture and the delta live tables. ...

Data Engineering
Reply
  • 868 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
john533
New Contributor III
  • 5 kudos

To synchronize data from an external database into Databricks with change data capture (CDC), you can use Delta Live Tables (DLT). Start by configuring a JDBC connection to your source database and use a CDC tool (like Debezium or database-native CDC...

  • 5 kudos
4 More Replies
GowthamR
by • New Contributor II
  • 255 Views
  • 2 replies
  • 0 kudos

Regarding Unity Catalog Self Assume Capabilities

Hi Team, Good Day! Recently in a Credentials section under Catalog , we have to add the self assume capabilities in the IAM role right.. Is it only for the Roles associated with Unity Catalog or for all the roles?. Thanks, Gowtham

Data Engineering
Reply
  • 255 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
RiyazAli
Valued Contributor II
  • 0 kudos

Hi @GowthamR,I believe it's only for the roles associated with UC. I was going through this community post on including self-assume capabilities for AWS IAM roles and it's mentioned that this change does not affect storage credentials that are not cr...

  • 0 kudos
1 More Replies
skanapuram
by • New Contributor II
  • 505 Views
  • 4 replies
  • 0 kudos

Error com.databricks.common.client.DatabricksServiceHttpClientException 403 Invalid access token

Hi I got this error "com.databricks.common.client.DatabricksServiceHttpClientException: 403: Invalid access token" during the run of a workflow job. It has been working for a while without error. Nothing has changed in regards to code or cluster. And...

Data Engineering
Reply
  • 505 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
john533
New Contributor III
  • 0 kudos

When the access token used for authentication is invalid or has expired, the error "com.databricks.common.client.DatabricksServiceHttpClientException: 403: Invalid access token" usually appears. Have you looked at the task cluster's driver logs? It m...

  • 0 kudos
3 More Replies
theron
by • New Contributor
  • 297 Views
  • 1 replies
  • 0 kudos

Liquid Clustering - Implementing with Spark Streaming’s foreachBatch Upsert

Hi there!I’d like to use Liquid Clustering in a Spark Streaming process with foreachBatch(upsert). However, I’m not sure of the correct approach.The Databricks documentation suggests using .clusterBy(key) when writing streaming data. In my case, I'm ...

Data Engineering
Reply
  • 297 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
RiyazAli
Valued Contributor II
  • 0 kudos

Hi @theron If you have enabled LC while table creation, you must have already specified the cluster column. Thus I don't see a reason to mention  .clusterBy(key) again.Let me know if any questionsCheers!If you want to create a brand new table with LC...

  • 0 kudos
David_Billa
by • New Contributor III
  • 480 Views
  • 1 replies
  • 0 kudos

Create delta table with space in field names per source CSV file

There are few fields in the source CSV file which has spaces in the field names. When I tried to create the table like `SOURCE 1 NAME` as string for few fields, I got an error message like 'INVALID_COLUMN_NAME_AS_PATH' error.  Runtime version is 10.2...

Data Engineering
Reply
  • 480 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
RiyazAli
Valued Contributor II
  • 0 kudos

Hi @David_Billa Before ingesting the csv data into the delta table, you could create delta table using the table properties as shown below:CREATE TABLE catalog_name.schema_name.table_name (`a` string, `b c` string) TBLPROPERTIES ( 'delta.minReade...

  • 0 kudos
lauraxyz
by • Contributor
  • 415 Views
  • 4 replies
  • 0 kudos

%run command: Pass Notebook path as a parameter

Hi team!I have a Notebook (notebook A) in workspace and I'd like to execute it with %run command from another Notebook (notebook B).  It works perfect with command: %run /workspace/path/to/notebook/ANow, i want to specify above path in a variable, an...

Data Engineering
Reply
  • 415 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

These global temp views are available to all workloads running against a compute resource, but they do not persist beyond the lifecycle of the cluster or the session that created them.You can refer to https://docs.databricks.com/en/views/index.html#t...

  • 0 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels