cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
naga_databricks
by • Contributor
  • 2959 Views
  • 2 replies
  • 1 kudos

Using Init scripts using DBX

I specify init scripts in my deployment.conf, as below: basic-static-cluster: &basic-static-cluster new_cluster: spark_version: "13.0.x-scala2.12" num_workers: 1 node_type_id: "n2-highmem-2" init_scripts: - worksp...

Data Engineering
Reply
  • 2959 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Naga Vaibhav Elluru​ We haven't heard from you since the last response from @Debayan Mukherjee​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be he...

  • 1 kudos
1 More Replies
AnkitG
by • New Contributor III
  • 6821 Views
  • 4 replies
  • 1 kudos

Resolved! Disable iFraming prevention not working

For my own workspace disable iFraming prevention stopped working on 9th May 2023. Till Friday it was working fine and I have already disabled the iFraming prevention.Is there something not right with the databricks platform?

Data Engineering
Reply
  • 6821 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Ankit Goyal​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...

  • 1 kudos
3 More Replies
ppang
by • New Contributor III
  • 5925 Views
  • 1 replies
  • 0 kudos

Resolved! Job cluster failed to start with custom docker image

Using Databricks Container Services, we have created two custom docker image: one based on nvidia/cuda:11.8.0-runtime-ubuntu22.04 and another based on databricksruntime/standard:12.2-LTS. In either case, we got this error with no specific diagnostics...

Data Engineering
Reply
  • 5925 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
ppang
New Contributor III
  • 0 kudos

I managed to get databricksruntime/standard:12.2-LTS to run in Databricks. However, for the CUDA image (nvidia/cuda:11.8.0-runtime-ubuntu22.04), I have only managed to get it to run with Databricks runtime 10.4 LTS. Does anyone know if Databricks run...

  • 0 kudos
JordanYaker
by • Contributor
  • 2968 Views
  • 1 replies
  • 0 kudos

Batch Doesn't Exist Failure

I have a job that's been working perfectly fine since I deployed it earlier this month. Last night, however, one of the tasks within the job started failing with the following error:java.lang.IllegalStateException: batch 4 doesn't exist at org.apac...

Data Engineering
Reply
  • 2968 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
JordanYaker
Contributor
  • 0 kudos

I tried FSCK REPAIR just on the chance that it would work and it had no effect.

  • 0 kudos
THIAM_HUATTAN
by • Valued Contributor
  • 1743 Views
  • 2 replies
  • 3 kudos

Resolved! Grant Databricks Access

In the above printscreen of Grant Databricks Access, we see we need to give the rights to a certain Bucket at the highest level. Why is this so? Are we able to limit the rights to only certain directories in a bucket, when we need Databricks to have ...

DatabricksQuestion
Data Engineering
Reply
  • 1743 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @THIAM HUAT TAN​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 3 kudos
1 More Replies
dsldan
by • New Contributor II
  • 6928 Views
  • 2 replies
  • 1 kudos

Resolved! DLT setup taking longer than actually building the tables

Hi all!We are using DLT for our ETL jobs, and we're noticing the setup steps (Initializing, Resetting tables. Setting up tables, Rendering graph) are taking much longer than actually ETL'ing the data into our tables. We have about 110 tables combined...

Data Engineering
Reply
  • 6928 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @daan duppen​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 1 kudos
1 More Replies
AJK1
by • New Contributor II
  • 1748 Views
  • 0 replies
  • 0 kudos

Identity Column Issues

Hello.I'm experiencing what I'm believe are pretty severe (current) shortcomings regarding Identity columns in Databricks.I'm defining a SQL table using spark SQL - the table creates as exepcted; I've tried using both column definitions for this iden...

CreateTable Insert-GeneratedAlways Command success image
Data Engineering
Reply
  • 1748 Views
  • 0 replies
  • 0 kudos
Moriondo
by • New Contributor III
  • 2147 Views
  • 3 replies
  • 2 kudos

Resolved! How to filter a dashboard by the current user email?

Hello,I would like to know if it is possible to filter a dashboard by the current user email?For example, I have a table result of a group of people with the following columns: user_id, user_email, date, productivity. So with this table I create som...

Data Engineering
Reply
  • 2147 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Moriondo
New Contributor III
  • 2 kudos

Hey guys, After some research on the documentation, I found out that if a filter the query using the current_user() function, I will get the result that I was looking for.If anyone need look at this:https://docs.databricks.com/sql/language-manual/fun...

  • 2 kudos
2 More Replies
Swaroop
by • New Contributor
  • 754 Views
  • 0 replies
  • 0 kudos

How to receive data from azure event hub in parquet ?

import asyncioimport osfrom azure.eventhub.aio import EventHubConsumerClientCONNECTION_STR = "Connection_string"EVENTHUB_NAME = "event_hub"async def on_event(partition_context, event):    # Put your code here.    # If the operation is i/o intensive, ...

Data Engineering
Reply
  • 754 Views
  • 0 replies
  • 0 kudos
Teja07
by • New Contributor II
  • 6692 Views
  • 0 replies
  • 0 kudos

Ingesting data from oracle to databricks through IICS

While ingesting the data from oracle to databricks through IICS, target table were created however data is not getting inserted. Below is the error. Could someone please help meException occurred when initializing data session. Root cause: java.lang....

Data Engineering
Reply
  • 6692 Views
  • 0 replies
  • 0 kudos
jannemanson
by • New Contributor III
  • 1701 Views
  • 3 replies
  • 0 kudos

Improve iteration time on implementing jobs

Hey there, I am using dbx to create Databricks tasks and deploy the job. I find it not ideal since the iteration circles are sometimes a bit long when I have to wait for a job with several tasks to complete and see where it failed. I am already tryin...

Data Engineering
Reply
  • 1701 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
jannemanson
New Contributor III
  • 0 kudos

Hello, thanks for the answer. Unfortunately, this did not help me, since it is general best practice. @Debayan Mukherjee​ 

  • 0 kudos
2 More Replies
de-hru
by • New Contributor III
  • 24872 Views
  • 4 replies
  • 1 kudos

Resolved! How to add pre-commit hook to the Git Client on Databricks Cluster?

I'd like to add a Git pre-commit hook to the Databricks Cluster.This pre-commit hook should be executed when pushing to GitHub.Why would I need a pre-commit hook on a Databricks Cluster?My goal is to run blackbricks and format all notebooks automatic...

Data Engineering
Reply
  • 24872 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Dejan Hrubenja​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels